ViewRenderable not being rendered initially

[Venthorus]

I have a weird problem with ViewRenderable not being rendered in one of my scenes and I wanted to ask if someone has faced the same problem at some point.

In my case I made my own SelectionVisualizer that uses a ViewRenderable. At first I suspected that selecting the node doesn't trigger the visualizer or changing the visualizer for the transformation system is not happening for some reason, but that was not the case (the original FootprintSelectionVisualizer works, however it uses a ModelRenderable).

At some point I then tried to add a simple node with a ViewRenderable and a normal button and the problem happened there too.

The problem resolves by leaving and then returning to the app, by rotating or after a permission request (so with other words: when onResume() is called), although it may take a considerable time before any node shows up again (even nodes with a ModelRenderable), but then everything shows up. I was debugging the app and as it seems the View of the ViewRenderable does not get attached to the window which is why in prepareForDraw() it doesn't get initialized. What makes me curious is that it can take a large amount of time before the nodes are visible again after resuming (can last from few seconds to dozens of seconds), but the plane itself is there immediately.

I didn't open an issue because it might be an issue on my side, however I can't wrap my head around why it is happening as it never happened to me before; I also don't understand what makes the affected scene so special that it only happens there. Does someone have any ideas?

[ThomasGorisse]

I have to say that it was the first time I opened the ViewRenderable class since there is not so much questions about it.
But you made an interesting inspection.

My first guess is that since the SelectionVisualizer is attached and detached from the parent node depending on the current selection, your issues could come from something related to the View reussing forbid like in a normal Activity layout.
Do you make a copy of your ViewRenderable each time the selected state changed?

Can you provide screenshots and make a PR about it?

[Venthorus]

I made a video that shows the problem:
https://user-images.githubusercontent.com/7419985/128327674-0413f237-9bf1-4c04-9365-f830a8b38bdd.mp4

Again, it happens with any ViewRenderable, not only those that are used as a SelectionVisualizer. So even if I add nodes with ViewRenderable, the nodes are there, but the View itself will only appear after onResume().

I never had this problem before; it only happens in this scene and I don't understand what makes this scene (or the architecture of the app?) so special that makes this issue occur.

It certainly might be an issue with the architecture of the app: I have a SceneFragment (which contains the UI you see in the video) and it can switch between AR and 3D, which means that the AR scene itself is a child fragment (which extends ArFragment) inside a FragmentContainerView. The ViewRenderable is created inside the child fragment at the end of onViewCreated(). I have a strong feeling that it must be some kind of race condition where something isn't ready yet so that a View of a ViewRenderable doesn't get attached to the window or something like that.

[RGregat]

I guess this is not easy to answer. I work by myself with the ViewRenderable class and I had similar issues as you have shown us. Most of the time it is really a timing issue and how you access the ViewRenderable. For me I implemented the following workflow.

In onViewCreated I prepare everything what I need for the start, for example a ViewRenderable to show where the camera ray is hitting the ground. I store the initialized ViewRenderable CompletableFuture in a HashMap for caching purposes. To access the ViewRenderable and attach it to a node I do something like this

CompletableFuture<ViewRenderable> completableFuture = mViewRenderableMap.get(renderableState).getFutureObject();
        CompletableFuture.allOf(completableFuture)
                .handle((notUsed, throwable) -> {
                    LoggerHelper.showLog(Log.DEBUG, TAG, "Call handle to get access to te ViewRenderables");
                    try {
                        ViewRenderable renderable = completableFuture.get();
                        renderable.setRenderPriority(Renderable.RENDER_PRIORITY_LAST);
                        renderable.setShadowCaster(isShadowCaster());
                        renderable.setShadowReceiver(isShadowReceiver());

                        if(verticalAlignment != null)
                            renderable.setVerticalAlignment(verticalAlignment);

                        getNode().setRenderable(null);
                        getNode().setRenderable(renderable);
                        LoggerHelper.showLog(Log.DEBUG, TAG, "Renderable replaced!");
                    } catch (InterruptedException | ExecutionException ex) {
                        ex.printStackTrace();
                    }
                    return null;
                });

This works just fine, I also tried to cache the ViewRenderable directly, but here I had sometimes issues where the ViewRenderables where not shown.

In a nutshell it is important to work with the CompletableFuture.

To help you better, we need atleast a code snipped.

[ThomasGorisse]

I add one last thing if the issue comes from your code (it's important for us to solve a founded ViewRenderable issue if there is):
Why are you using a ViewRenderable for your SelectionVisualizer? If you just want to display a circle with 4 direction triangles around, you should definitely use a glTF instead.
Blender is not so complicated and after some beginners tutorials, you will be able to use it just like you use Gimp or Photoshop to generate/modify your app images.

[Venthorus]

Well, I guess the reason is that it worked without problems so far. I use ViewRenderable fairly often and I'm facing this problem the first time.

Of course in this case I can switch to ModelRenderable, but it is a problem nonetheless (at least it seems so), so it would be interesting to see what the root of the problem is. I will probably take another look at it in the coming days.

The code is trivial; at the end of onViewCreated() the following code is executed (code is in Kotlin):

ViewRenderable.builder()
  .setView(requireContext(), R.layout.selection_visualizer_view)
  .setVerticalAlignment(ViewRenderable.VerticalAlignment.CENTER)
  .build()
  .thenAccept { renderable ->
    transformationSystem.selectionVisualizer = RenderableSelectionVisualizer(renderable)
  }

I think the code itself is not necessarily the problem, I used it many times. Again it also happens with normal nodes with ViewRenderable as a renderable; it is not specific with the selection visualizer.

Something else seems to interfere with the View of ViewRenderable not being attached to the window. As @RGregat already mentioned it seems to be a timing issue (I guess it can't be a garbage collection issue because then it probably wouldn't work after onResume()).

[Venthorus]

I found the culprit and I "fixed" it for myself, but I still don't understand why it is happening in the first place.

The problem starts with the fact that the View does not get attached to the window, so in ViewRenderable.prepareForDraw() the code does not pass the following check:

if (!renderViewToExternalTexture.isAttachedToWindow()
    || !renderViewToExternalTexture.isLaidOut()) {
  // Wait for the view to finish attachment.
  return;
}

Only if it passes this check the ExternalTexture will be set. So the question is why the view does not get attached. The problem is in ViewAttachmentManager.onResume():

void onResume() {
  // A ownerView can only be added to the WindowManager after the activity has finished resuming.
  // Therefore, we must use post to ensure that the window is only added after resume is finished.
  ownerView.post(
      () -> {
        if (frameLayout.getParent() == null && ownerView.isAttachedToWindow()) {
          windowManager.addView(frameLayout, windowLayoutParams);
        }
      });
}

The Runnable simply does not run when called the first time for some reason. I "fixed" it by calling this method in onResume() of my Fragment:

arSceneView.renderer?.onResume()

The Runnable gets executed and so each ViewRenderable works as expected. This "fix" shouldn't have any side effects, as it pretty much does nothing after the View is attached.

I don't understand what is going on here. Is something else removing the Runnable from the message queue or is it a performance/race condition issue? Never seen something like this before.

Well at least I can continue using my good old ViewRenderable.

[ThomasGorisse]

Nice job.

We (@RGregat and I) have worked on many things including some resume/pause/destroy fixes here: #143

I have made tests with it and, in my environment, the Runnable seems to be called immediately after the post in the onResume() in here:

ownerView.post(
      () -> {
        if (frameLayout.getParent() == null && ownerView.isAttachedToWindow()) {
          windowManager.addView(frameLayout, windowLayoutParams);
        }
      });

Could you checkout the Pull Request and check that this issues is fixed with it ?

[ThomasGorisse]

The pull request has been included in the 1.19.6.
@Venthorus Can you please check it out ?

[Venthorus]

It sadly doesn't fix it. My fix is still required (which is okay for now).

It actually introduces a bug where when the ViewRenderable is rendered for the first time it shows a frame of the fireplace video which is used in the VideoNode inside the chimney (in my video I posted earlier you can barely see it).

I suppose ViewRenderable also uses ExternalTexture underneath so there seems to be an issue with it. If I remove the VideoNode the first frame shows some gibberish from the surrounding area.

[ThomasGorisse]

Thanks for your great feedback!

I add you to the contributors, can you make a quick PR with your fix?

For the flickering/wrong texture appearing, I think it's related to #148
@RGregat My thought goes more now to a wrong filamat generation from the Camera and VideoNode

[RGregat]

I stumbled across this topic again and found an old issue on the original Sceneform Repository. Unfortunately no solution is described here, but the issue must be the same as we have currently with the wrong texture for ViewRenderables for a couple of frames. I'm not sure what I should do with the new information from the other repository, but I thought it might me interesting for the rest here.