sceneview-android: Augmented Face - Not Rendering Model and Still in "surface mode"

Hello, I was trying to understand learn and understand “sceneview” usages, I already discovered how to get Augmented face in “sceneform-android” but when I try sceneview, I encountered several problems,

  • First camera settings are seems to be not working properly because some options act wrong. (For example MESH3D mode or I was able to enable Deph mode (normally I shouldn’t be able to do that in Front Camera) )
  • Model size is too small
  • Model moving opposite side when I move my head (If I go right, model goes left)
  • Model not render properly and shows me “black material”
  • I’m always seeing “Searching for surfaces…” (front camera or back camera) It should not be visible in front camera

Here my minimal code (only MainActivity)


    private lateinit var binding: ActivityMainBinding
    private val facesNodes = HashMap<AugmentedFace, AugmentedFaceNode>()
    private var faceModel: ModelRenderable? = null
    
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        setContentView(binding.root)
        loadModels()

    }
    
    private fun loadModels() {

        lifecycleScope.launchWhenCreated {
            binding.sceneView.configureSession { arSession, config ->
                val filter =
                    CameraConfigFilter(arSession).setFacingDirection(CameraConfig.FacingDirection.FRONT)
                val cameraConfig = arSession.getSupportedCameraConfigs(filter)[0]
                arSession.cameraConfig = cameraConfig
                config.instantPlacementMode = Config.InstantPlacementMode.DISABLED

                config.augmentedFaceMode = Config.AugmentedFaceMode.MESH3D
                arSession.configure(config)
            }


            faceModel = loadModel(
                context = this@MainActivity,
                lifecycle = lifecycle,
                glbFileLocation = "models/fox.glb",
            )

            binding.sceneView.onAugmentedFaceUpdate = {
                val existingFaceNode = facesNodes[it]
                when (it.trackingState) {
                    TrackingState.TRACKING ->
                        if (existingFaceNode == null) {
                            val faceNode = AugmentedFaceNode(binding.sceneView.lifecycle, it)
                            val modelInstance = faceNode.setFaceRegionsRenderable(faceModel)
                            modelInstance.isShadowCaster = false
                            modelInstance.isShadowReceiver = true
                            faceNode.faceRegionsRenderable = faceModel!!
                            binding.sceneView.addChild(faceNode)
                            facesNodes[it] = faceNode
                        }
                    TrackingState.STOPPED -> {
                        if (existingFaceNode != null) {
                            binding.sceneView.removeChild(existingFaceNode)
                            facesNodes.remove(it)
                        }
                        facesNodes.remove(it)
                    }
                    else -> {
                        facesNodes.remove(it)
                    }
                }
            }

        }
    }

activity_main.xml snippet as well

    <io.github.sceneview.ar.ArSceneView
        android:id="@+id/sceneView"
        android:layout_width="0dp"
        android:layout_height="0dp"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

PS : I used "fox and “face” glb files from sceneform for quick testing.

If you need additional information please let me know.

Thank you.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (7 by maintainers)

Most upvoted comments

Hello I find the solution, you need to put this after config change

sceneView.onArSessionResumed = { it.setCameraTextureNames(sceneView.arCameraStream.cameraTextureIds) }

It is the same issue as in: https://github.com/google-ar/arcore-android-sdk/issues/1170#issuecomment-825339396

I think that explanation there is a little bit wrong. The texture ID points to the OpenGL texture but it is up to ARCore to actually create this texture and fill it with camera frames. When setTextureName is called we connect the texture with the camera that is currently specified by the ARCore CameraConfig. Therefore, the texture and camera IDs don’t have to match. Moreover, when using multiple textures with setTextureNames each texture from the array has its own ID.

It can be so that calling setCameraConfig resets the texture names and we need to set them again, for example, by setting hasSetTextureNames to false.

For my end, I had a busy month and it is getting close to finish. I will come up with update and code change, for texture problem I had to check so I can’t answer that one atm

@ThomasGorisse

I added new branch with AugmentedFaceNode -> new branch , you may check it out

I hope that’s alright. It’s still wip obviously because I am still in the process of understanding new SceneView in general.

Some of the issue here:

  • usage of Ar, Model or basic Node? Evene regular node doesn’t have renderable setter?
  • do we need augmentedImageDatabase for saving frames?
  • are we using old Engine(Instance) for transformation? And Matrix from Sceneform.math since I haven’t found one from Romain Guy?

Just few questions for start

  • That’s because (afaik) you need to add those in “sceneview-extended” because in node folder all I saw was video and image nodes rest of them wasn’t exist yet.
  • you don’t need AugImageDatabase
  • https://github.com/romainguy/kotlin-math
  • For the Augmented Face stuff you gonna need to handle “occlude” material side as well (because pure augmented face from arcore doesn’t hide anything behind your face or head

@onuralpszr

Have you had any luck in fixing this issue?

I made bit more progress on this and trying to stable somehow so I can share with PR and repo so everybody try it (of course try with care, it can be break 😃) )