mediapipe: ImageSegmenter creation from Web Worker hangs on iOS

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

iOS 16.5.1

Mobile device if the issue happens on mobile device

iPhone X

Browser and version if the issue happens on browser

Safari 16.5.1, Chrome 115.0.5790.160

Programming Language and version

JavaScript

MediaPipe version

0.10.3

Bazel version

No response

Solution

ImageSegmenter

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

When the ImageSegmenter is created in a Web Worker on iOS Safari or iOS Chrome, createFromOptions never returns, or at least, it doesn't return in under 60 seconds.

When run on the main thread, createFromOptions works as expected on iOS.

Describe the expected behaviour

Since ImageSegmenter.createFromOptions works fine from Web Workers on Chromium browsers and Firefox on Windows, and it works fine on all platforms tested when not using a Web Worker, I would expect it to also work on iOS under Web Workers, or at least get some error message, or at the very least, for this to be a current, documented limitation.

Standalone code/steps you may have used to try to get what you need

Relevant code:


    initializeAsync() {
        if (!this._isInitialized && !this._isInitializing) {
            this._loc = 1;

            this._isInitializing = true;
            let taskVision;
            const module = 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3';
            import(module)
                .then((result) => {
                    this._loc = 2;

                    taskVision = result;
                    return taskVision.FilesetResolver.forVisionTasks(
                        'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3/wasm'
                    );
                })
                .then((vision) => {
                    this._loc = 3;

                    const options = {
                        baseOptions: {
                            modelAssetPath:
                                'https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite',
                            delegate: 'CPU'
                        },
                        runningMode: 'VIDEO',
                        outputCategoryMask: true,
                        outputConfidenceMasks: false
                    };
                    return taskVision.ImageSegmenter.createFromOptions(vision, options);
                })
                .then((segmenter) => {
                    this._loc = 4;

                    this._imageSegmenter = segmenter;
                    this._isInitialized = true;
                    this._isInitializing = false;
                });
        }
    },


### Other info / Complete Logs

_No response_

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 19

Most upvoted comments

Thanks for the explanation. iOS 17 is released today, it seems, so that ‘temporary’ might be very temporary indeed… here’s hoping. [Edit: OffscreenCanvas with webgl works, now].

Edit: The Safari 17 Release Notes | Apple Developer Documentation say:

  • Added support for 3D OffscreenCanvas WebGL support. (39882956)

Let me re-open this.

While Safari now supports OffscreenCanvas, it does not yet support WebGL2RenderingContext: https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas/getContext. We are blocked on this right now.

Actually I came across that fix while hunting for ideas, but I do not think this will fix this particular problem.

This access of the DOM happens on this line of code:

https://github.com/google/mediapipe/blob/master/mediapipe/tasks/web/vision/core/vision_task_runner.ts#L52

As you can see from a couple lines above that, this is a problem on WebKit.

Moreover, my kludge of passing in an OffscreenCanvas as stated above only works on iOS, but not iPadOS, as I eventually found out. Hence I believe that as it stands, using MediaPipe Tasks from a Web Worker in WebKit will just not work, generally speaking, as this issue is unrelated to the platform_utils isIOS() function.

I added a catch() to my Promise cascade (should have done this earlier) and I get:

ReferenceError: Can't find variable: document

Big clue 💡

I think the problem lies at the very beginning of createVisionInstance(), specifically in createCanvas().

On the iPhone, isWebKit() will be true and then we will try to access the DOM … from a Web Worker.

I’m not sure what a proper solution can be here. It looks like the caller can pass in their own canvas, but I don’t know what properties that canvas should have or what effects on performance it would even have. I can’t find any mention of the canvas option in the MediaPipe docs.