mediapipe: FaceMesh nose landmarks not correct on artificial (rendered) facial imagery

Please make sure that this is a bug and also refer to the troubleshooting, FAQ documentation before raising any issues.

System information (Please provide as much relevant information as possible)

Have I written custom code (as opposed to using a stock example script provided in MediaPipe): to some extent custom code. But it’s mostly based on the example code provided by MediaPipe for FaceMesh on Python.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Windows 11
Programming Language and version ( e.g. C++, Python, Java): Python 3.9
MediaPipe version: 0.8.9.1
Solution ( e.g. FaceMesh, Pose, Holistic ): FaceMesh

Describe the current behavior: On artificial (rendered) facial imagery, nose width does not show as much impact on land marks as other facial attributes

Describe the expected behavior: On artificial (rendered) facial imagery, nose width should be reflected correctly in the detection of facial landmarks

Standalone code to reproduce the issue: I’ve added a couple example images that show the issue. Raw images, images with rendered landmarks, combined images showing the two extremes (e.g. nose width blendshape of the digital character set to -1 and +1, same about nose height, mouth width and eyes height)

Additional information Since the model provides valid detection of almost all landmarks with all blendshapes I’ve tested, except for the nose, I’ll consider this a bug, even though the model assumedly isn’t really made for detecting landmarks on artificial data.

I’d appreciate any help. Maybe there’s something I can do with the rendered image (shaders…) to achieve renders that MediaPipe will work better with?

Information about the attached files

All files have the corresponding blendshape in the filename (chin_height etc.). m1 means blendshape value set to -1, p1 means blendshape value set to +1. The delta png files contain renderings of the landmarks only. The lm files contain the faces plus the detected landmarks. The files without landmarks can be used to reproduce the issue by means of the MediaPipe example code.

As you can see below, the different blendshapes all have at least some impact on the landmarks. Except for the nose width blendshape where there’s almost no noticable difference (red and green landmarks around nose match, regardless of the actual width of the nose) between nose width = -1 or nose width = +1.