Research: XR Support

[vanruesc]

Description

Related: #676 #452

This is a research ticket for gathering information on how to implement XR support in postprocessing.

The rendering workflow in v7 consists of using a RenderPipeline that uses a WebGLRenderer/WebGPURenderer to render individual passes. Three's WebXR support requires the use of WebGLRenderer.setAnimationLoop and it uses a special XR camera for the active XR session. So some parts definitely need to be changed but the actual scope of the changes needs to be specified before VR can be implemented.

Implementation Details

• TBD

Relevant sources

• Unregister the system / unload effects #216 (comment)
• Post-processing with WebXR mrdoob/three.js#18846 (comment)
• Post-processing with WebXR mrdoob/three.js#18846 (comment)
• WebGLRenderer: Improve offscreen rendering in WebXR. mrdoob/three.js#26160
• https://threejs.org/docs/?q=vr#manual/en/introduction/How-to-create-VR-content
• https://threejs.org/docs/?q=xr#api/en/renderers/webxr/WebXRManager
• https://developer.mozilla.org/en-US/docs/Web/API/WebXR_Device_API/Rendering

Additional sources

• https://github.com/orgs/angular-threejs/discussions/7
• https://web.archive.org/web/20220804162612/https://twitter.com/Cody_J_Bennett/status/1482585611781480448
• https://codesandbox.io/p/sandbox/webxr-postprocessing-forked-8hzc9l?file=%2Fsrc%2Findex.js
• Adding support for WebXR react-postprocessing#128

[issimissimo]

I'm trying to make it work with the 6.x version in AR app, but no luck.
I've Just added the RenderPass as a test, but meshes are not displayed anymore
... composer = new POSTPROCESSING.EffectComposer(renderer) composer.addPass(new POSTPROCESSING.RenderPass(scene, camera)); ... composer.render()

I don't think that setAnimationLoop is the real issue, but xr camera.

So, I'm guessing, what about to use another standard scene, with another standard camera "linked" to the xr camera, and use the render texture onto a plane in front of the xr camera?
For VR probably it's useless because of the perspective, but maybe for AR it could work...

[vanruesc]

I'm trying to make it work with the 6.x version in AR app, but no luck.

It's not supported in either v6 or v7 at this time. This ticket is about the planned implementation in v7 only.

I don't think that setAnimationLoop is the real issue, but xr camera.

It's definitely relevant to the issue:

• https://threejs.org/docs/?q=renderer#api/en/renderers/WebGLRenderer.setAnimationLoop
• https://threejs.org/manual/#en/how-to-create-vr-content

[issimissimo]

Let me explain:
1 - I'm using V6, so I'm talking about it right now.
I'm using both "setAnimationLoop" and "requestAnimationFrame" with "postprocessing" in a desktop web app, and both them work.
That's why I had idea that there should be no issue on it for XR, if "setAnimationLoop" is mandatory, and that's why I suspect that the problem is about the xr camera.
2 - By now, my suggestion would be to use another canvas, with another renderer, to suverimpose the "postprocessing" render onto the top of the underliying scene, so to be able to use a standard camera for rendering, instead of the xr camera.
A bit tricky, but I did a test and it work.
Next step is to sync the standard camera with the xr camera, but it's not so hard.
Maybe I'm wrong, or maybe it can give you some idea.
Btw many thanks!!!

[vanruesc]

I'm using V6, so I'm talking about it right now.

I see. This ticket is about v7 and there are no plans to add XR support to v6. Sorry for the confusion - I've added the v7 milestone now.

that's why I suspect that the problem is about the xr camera

Well, yes, no question about that. Postprocessing needs to be aware of the array camera and XR state.

my suggestion would be to use another canvas, with another renderer [...]
Next step is to sync the standard camera with the xr camera, but it's not so hard.

There is no need for an additional renderer. Postprocessing can use the same renderer since it uses its own internal render targets and doesn't actually use a camera to render the effects. (The fullscreenCamera is just a dummy.)

to superimpose the "postprocessing" render onto the top of the underliying scene, so to be able to use a standard camera for rendering, instead of the xr camera.

With XR, the scene is rendered twice - once for each eye, with slightly different camera settings and the results are rendered into the left half and the right half of a framebuffer respectively. Postprocessing must again be aware of this and also render the effects for the left eye and the right eye while respecting the different camera settings. Blending the result with the XR output frame is not the problem.

I appreciate your input and I hope this clears things up a bit.

[issimissimo]

Yes, I see. Many thanks for your nice and detailed reply.