Native media Equirect layers creation

[danrossi]

Related issue: #31029

Description

Adding native media layer creation for WebGPU XR. This fixes a context loss issue doing it outside the manager. There is a potential Oculus browser bug complaining makeXRCompatible isn't set when setting up a media layer still.

Media layers are prepended into the background before the other layers.

Example https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	359.27 85.31	359.27 85.31	+0 B +0 B
WebGPU	630.13 174.94	632.45 175.85	+2.31 kB +904 B
WebGPU Nodes	628.72 174.69	631.03 175.6	+2.31 kB +903 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	491.03 119.72	491.03 119.72	+0 B +0 B
WebGPU	703.7 189.99	706.02 190.88	+2.32 kB +893 B
WebGPU Nodes	652.93 177.41	655.25 178.37	+2.32 kB +956 B

[danrossi]

It took a bit to design how to go about it. The only issue is you have to toggle the video mesh layers when going into XR and ending the session. Maybe there is other ways to do it.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L130
https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L151

live example https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

[Mugen87]

I have tried to improve the JSDoc a bit. For a functional review I hope @cabanier can have a look.

BTW: Please create a E2E screenshot for the new example webgpu_xr_media_layer. You can do that with npm run make-screenshot webgpu_xr_media_layer.

[danrossi]

looks ok

[danrossi]

The requirement to toggle off the video texture with the meshes. I'm not sure how to make better. It causes a crash if the texture is also rendered.

[cabanier]

Please match behavior of quad/cylinder layers

[danrossi]

I've refactored it quite a bit. And followed the structure of modifying the geometry externally to keep that external. I could add that in. It returns a group of meshes which works in 2D, Rift, Quest.

There is still the problem of needing to turn off meshes for native layers. Can be done internally or keep it external. I'm not across everything to figure out a better way to render from 2D to native.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

mono example https://electroteque.org/dev/threejs/examples/webgpu_xr_media_mono_layer.html

[danrossi]

Is this looking good setting the geometry externally.

I have it working implementing and tested. Unfortunately media layer needs to be added in the manager as doing it external caused a context loss fault. It may be a problem adding other layers externally as the media layers list needs to be prepended in the list as backgrounds as I have done.

The only issue still I can't figure out when going from the 2D mesh to native layer in XR. It will be rendered including the video texture in the projection layer as its added to the scene so those layers need to be turned off. Or maybe simply removing those mesh groups from the scene ?. I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer.

[cabanier]

... doing it external caused a context loss fault.

Do you have an example for me to look at? You shouldn't be getting context losses.

I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer.

Yes, can't you do the same as what I did for quad/cylinder layer?

[danrossi]

looks ok

[danrossi]

If the external geometry setter is ok. Or I make them default SphereGeometry requiring to update it's settings. I just set them to PlaneGeometry.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L105

I add the mesh group to the layers params.

https://github.com/danrossi/three.js/blob/equirect-layers/src/renderers/common/XRManager.js#L667

So this requirement to turn off texture rendering in the projection layer can be done internally perhaps and make it transparent.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114

There is an issue I can't figure out if it's the browser or three. Native media layer rendering is blank if it's an immersive-vr session. Only works in immersive-ar session. The controllers don't connect all the time either and can't figure out where to report. There is still the makeXRCompatible error in the console I can't find where to report.

[danrossi]

It might need a blank render target like the other layer examples. When using immersive-vr mode instead of immersive-ar mode like the XRButton uses. The projection layer render target blocks out the background native media layer. So it seems it's a blank render.

https://github.com/mrdoob/three.js/blob/dev/src/renderers/common/XRManager.js#L1480
https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl-fallback/WebGLBackend.js#L354

https://electroteque.org/dev/threejs/examples/webgpu_vr_media_layer.html

Doing something like this in the renderLayers method seems to work but not sure if its worth it and just keep it an immersive-ar session. If it peforms the same. Is just to be able to not have the projection layer render target cover it up. I noticed all the demos show up weird because they are AR sessions though. So stuff around the camera.

for (const layer2 of this._mediaLayers) {
    layer2.renderTarget.isXRRenderTarget = this._session !== null;
    layer2.renderTarget.hasExternalTextures = 
    layer2.renderTarget.isXRRenderTarget;
    layer2.renderTarget.autoAllocateDepthBuffer = ! 
    layer2.renderTarget.isXRRenderTarget;

    if ( layer2.renderTarget.isXRRenderTarget && this._supportsLayers ) {

        const glSubImage = this._glBinding.getSubImage( layer2.xrlayer, this._xrFrame );
	this._renderer.backend.setXRRenderTargetTextures(
		layer2.renderTarget,
		glSubImage.colorTexture,
		glSubImage.depthStencilTexture );

    }

}

[danrossi]

I was able to figure out fixing the conflict in the editor but have been testing on a merged branch. I'm still unable to figure out how to work around three projection layer render target hiding the native layer in immersive-vr mode. Only works in immersive-ar mode. Which is what all the XR examples use.

Hopefully disabling the 2D mesh layer internally to not render in the projection layer is the right solution.

[danrossi]

I found the layer index needs to be set to 0 to support mono video or else only one eye is rendered in XR. Since a three update.

[danrossi]

Everything is stabilised and operating correctly. This feature is needed in Quest if using multiview due to a bug #32151 I reported using multiview with video textures. It causes flickering. So for Quest native media layer can be used instead.

[danrossi]

I just implemented 2D Depth Stereo 3D Video support but still yet to produce a demo video due to huge resources required. It's like the stereo render but with planegeometry used instead and shows a Cinema screen display. The picture looks 3D as AI extracts depth and adds it to the right eye. A fisheye distortion can be added also that may work with 180 stereo video rendering layer instead.

Upon discovery there is a plane video layer api for video which will help render this 2D 3D format ! So the same layout options for stereo and mono. I will look at integrating this into the same method now.

https://developer.mozilla.org/en-US/docs/Web/API/XRMediaBinding/createQuadLayer

[danrossi]

I have more changes coming to support quad video layers. Which is working but has distance and scaling issues. It required refactoring mesh creation for the other layers into a utility module to remove duplication which I tested the other layer demo is working. There is standard modules for sphere and plane mesh creation and the blend mesh creaction.

[danrossi]

There is now support for mono/stereo quad layers. With resusable utility methods for creating mono/stereo meshes. The other layers have been refactored with the reused blending and mesh creation to reduce duplication. It took alot of trial and error the Z transform with native layers needs to be targeted and an offset of 2 meters is needed for non layers so it shows the same position which is being done internally. Native layers shows closer otherwise.

I just noticed the right eye mesh can reuse the material of the left eye. And the blend material can be swapped out in place of the textured material.

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html?layout=quad-stereo

[danrossi]

I ran into a strange bug with media quad layers. The transform units for Z, double is required to the world units. Which is transforming the blend mesh layer. So the translation stays in world units and needs to be doubled for the native layer for Z anyway. That feature is working.

https://github.com/danrossi/three.js/blob/equirect-layers/src/renderers/common/XRManager.js#L788

[danrossi]

I added the media layer feature almost 12 months ago. Is it possible to check if the design is correct and if useful. Its stable for me now. But for WebGPU support XRManager requires a factory method to handle layers like the WebGPURenderer system.