mediapipe: Cannot run selfie segmentation in web worker

Please make sure that this is a solution issue.

System information (Please provide as much relevant information as possible)

  • Have I written custom code (as opposed to using a stock example script provided in Mediapipe): N/A
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Any
  • MediaPipe version: 0.1.1632777926
  • Bazel version:
  • Solution (e.g. FaceMesh, Pose, Holistic): Selfie Segmentation
  • Programming Language and version ( e.g. C++, Python, Java): JavaScript

Describe the expected behavior:

Solution works the same way as in main thread

Standalone code you may have used to try to get what you need :

I was using Webpack 5.x and npm install @mediapipe/selfie_segmentation.

// main thread
const worker = new Worker(new URL('./worker.ts', import.meta.url));

// worker.js
import { SelfieSegmentation } from '@mediapipe/selfie_segmentation';

const ss = new SelfieSegmentation({
    locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/${file}`,
});

ss.setOptions({
    modelSelection: 1,
});

export const processFrame = (image: ImageBitmap) =>
    new Promise((resolve, reject) => {
        ss.onResults((results) => {
            ss.onResults(null);
            resolve(results.segmentationMask);
        });
        ss.send({ image }).catch(reject);
    });

Other info / Complete Logs :

Looks like some DOM objects are used in the solution code:

Uncaught ReferenceError: document is not defined
    at new Xb (selfie_segmentation.js?1643:73:229)

Additionally, ss.send has type issue:

TS2322: Type 'ImageBitmap' is not assignable to type 'InputImage'.   Type 'ImageBitmap' is missing the following properties from type 'HTMLCanvasElement': captureStream, getContext, toBlob, toDataURL, and 280 more.  index.d.ts(19, 3): The expected type comes from property 'image' which is declared here on type 'InputMap'

Similar bugs:

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 30

Most upvoted comments

@kirill-konshin,

Our MediaPipe Tasks can run in web worker. Please revert back if you still encounter the similar error.

No it cannot.

Error in worker: Error: Failed to execute ‘importScripts’ on ‘WorkerGlobalScope’: The script at ‘http://localhost:3000/_next/static/media/selfie_segmentation_landscape.eec7c3d6.tflite’ failed to load.

Version 0.1.1675465747 still requests tflite file via importScripts. It has to do fetch instead.

                    if ("function" !== typeof importScripts) throw Error("solutions can only be loaded on a web page or in a web worker");
                    d = b.filter(function(n) {return void 0 === n.simd || n.simd && c || !n.simd && !c}).map(function(n) {return a.locateFile(n.url, a.ha)});
                    importScripts.apply(null, ea(d));
+2
kirill-konshin on Jun 22, 2023

Could be approach use the model directly on a web worker using tensorflow?

importScripts('https://cdnjs.cloudflare.com/ajax/libs/tensorflow/4.2.0/tf.min.js');

async function loadModel() {
  const model = await tf.loadGraphModel('https://storage.googleapis.com/mediapipe-assets/selfie_segmentation.tflite');
  return model;
}

let modelPromise = loadModel();

onmessage = async (event) => {
  const { imageData } = event.data;
  const tensor = tf.browser.fromPixels(imageData);
  const model = await modelPromise;
  const output = await model.predict(tensor.expandDims(0)).squeeze();
  const segmentationMask = output.gather([0], 2).greater(0.5);
  const segmentationData = await segmentationMask.data();
  const width = segmentationMask.shape[1];
  const height = segmentationMask.shape[0];
  postMessage({ segmentationData, width, height });
};
+1
matiaslopezd on Mar 14, 2023

Worker thread has no access to main thread, so this will not work. The problem is that these files are loaded via importScript instead of fetch.

I suppose it would be hard to pass the object around. But importScripts should work with workers. It is only the mime type problem. I think for one have you tried to host your model file with modified types to bypass the mime type check? Because I tried to host the tflite model at localhost and changed the filename to js (also need to modify the file path in mediapipe’s package) and then the error is gone. So it has no problem import scripts. Finally, I still don’t see the reason for the onresult to be put in the worker as it’s completely non-blocking. The official example didn’t even await for initialize().

+1
cy-moi on Nov 7, 2022

Hi, I’m having the same issue with face_mesh 0.4.1633559619. Have you figured it out?

Update: I removed the && "ontouchend" in document at line 75 in face_mesh.js and it worked.

+1
cy-moi on Sep 5, 2022
Read more comments on GitHub
← mediapipe: C++ compilation of rule '//mediapipe/calculators/tensor:image_to_tensor_converter_opencv' failed (Exit 1)
mediapipe: Can't install mediapipe on PyCharm IDE. →