GaussianSplats3D: Incorrect Rendering results on Apple Vision Pro

I viewed GaussianSplats3D in WebVR mode on AppleVisionPro. It generally works for the areas where splats are present, but did not work well for the areas where splats are not present.

Video of the areas where splats are present. https://twitter.com/kotauchisunsun/status/1758524300318191932

Video of the areas where splats are not present. https://twitter.com/kotauchisunsun/status/1759187480174305319

Here is the source code.

<html lang="en">

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="x-ua-compatible" content="ie=edge">
  <title>3D Gaussian Splat Demo - VR Garden</title>
  <script type="text/javascript" src="js/util.js"></script>
  <script type="importmap">
    {
        "imports": {
            "three": "./lib/three.module.js",
            "@mkkellogg/gaussian-splats-3d": "./lib/gaussian-splats-3d.module.js"
        }
    }
  </script>
  <style>

    body {
      background-color: #000000;
      height: 100vh;
      margin: 0px;
    }

  </style>

</head>

<body>
  <script type="module">
    import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';
    import * as THREE from 'three';
    const viewer = new GaussianSplats3D.Viewer({
      'initialCameraPosition': [-0.77493, -0.54359, 1.63199],
      'initialCameraLookAt': [0.20786, -0.68154, -0.27311],
      'webXRMode': GaussianSplats3D.WebXRMode.VR,
      'gpuAcceleratesSort': false,
      'sharedMemoryForWebWorkers': false
    });
    let path = 'assets/data/bonsai/bonsai.ksplat';
    viewer.addSplatScene(path, {
        'rotation': new THREE.Quaternion().setFromUnitVectors(new THREE.Vector3(0.01933, -0.75830, -0.65161).normalize(), new THREE.Vector3(0, 1, 0)).toArray(),
        'scale': [0.25, 0.25, 0.25],
        'position': [0, 1, 0]
    })
    .then(() => {
        viewer.start();
    });
  </script>
</body>

</html>

Thanks for the nice library!

About this issue

Original URL
State: closed
Created 4 months ago
Comments: 19 (10 by maintainers)

Most upvoted comments

The latest release v0.3.9 now has this fix in it 😃

mkkellogg on Apr 22, 2024

So I have a WIP branch that I think addresses the issue (at least in the WebXR emulator extension): https://github.com/mkkellogg/GaussianSplats3D/tree/fix_webxr_stereo. Can you try that out and let me know if it works for you?

It works perfectly on a Quest 3!

JiamingSuen on Apr 18, 2024

So I think the issue is that three.js (and my viewer) can’t tell when WebXR is rendering in stereo mode, and therefore don’t know that the aspect ratio needs to be adjusted in that case. Basically when rendering in stereo mode, the viewport width value that is sent to the shader for the splat renderer needs to be halved. I’ll keep digging!

mkkellogg on Apr 17, 2024

I will take a look at the stereo rendering bug, hopefully it’s an easy fix. As for the hard alpha falloff, I’m guessing it might have something to do with the fact that in my viewer the splats are rendered as quads that are aligned to the splat’s 2D screen-space axes that are clipped at sqrt(8) standard deviations. I try to document that in the shader code. The value sqrt(8) may seem somewhat arbitrary; I’m mainly using it because it is (less obviously) the same cutoff that is used in other online viewers like Antimatter’s splat viewer and the Unity Gaussian splat viewer You could try increasing that value here: https://github.com/mkkellogg/GaussianSplats3D/blob/0f5b839b4f6d30d5c65b487b20844f4b2cad61e2/src/SplatMesh.js#L157 and then its corresponding squared value here: https://github.com/mkkellogg/GaussianSplats3D/blob/0f5b839b4f6d30d5c65b487b20844f4b2cad61e2/src/SplatMesh.js#L357 (I probably need to have better variable names 😃 )

In the metal-splats repo they use the original approach to rendering, which is calculate a bounding box that is aligned to the screen and has a side length that is the maximum of the 2 screen-space eigen values scaled by 3.0, which means at least 3.0 standard deviations: https://github.com/laanlabs/metal-splats/blob/4a1c3f05c06a3ddf5f7358d6aaadd889f3e47d64/MetalSplat/SplatShaders.metal#L285. This means their bounding box is less compact than the one I calculate in my viewer, and maybe that’s why they don’t have a hard alpha fall-off issue.

I would also guess that the gamma issue you mentioned might be exacerbating the issue in my viewer. However this is all just a guess! I will investigate further and get back to you once I figure something out.

mkkellogg on Apr 9, 2024