cordova-plugin-iosrtc: VideoElement cannot play video stream

Hello,

Recently I tried to rewrite an iOS SDK which invoke browser-based WebRTC APIs internally to invoke ‘cordova-plugin-iosrtc’ APIs internally. By now, some API works fine like I can join channel using RTCPeerConnection and publish local video stream using getUserMedia to a peer browser. Amazing plugin!

But <video> play cannot work, the <video> viewport always show its black background but no video stream.

My html look like this

<body>
    ...
        some other tags 
    ...    
    <video id="local-video" muted autoplay="autoplay" playsinline="playsinline" webkit-playsinline _src="assets/Sunset.mp4" style="width:480px; height:360px; background-color:black;">
    </video>
</body>

My js look like this

window.navigator.mediaDevices.getUserMedia({
        video: {
            deviceId,
        },
        audio: true
    }).then(stream => {
        console.log(`getUserMedia.stream = ${JSON.stringify(stream)}`);
        console.log(`getUserMedia.stream.getTracks = ${JSON.stringify(stream.getTracks())}`);

        // localVideoEl = document.querySelector('.local-video');
        localVideoEl = document.getElementById('local-video');

        // cordova.plugins.iosrtc.observeVideo(localVideoEl)

        localStream = stream;

        // Listen to all events
        TestListenEvents(localStream);
        TestListenEvents(localVideoEl);

        // Attach local stream to video element
        localVideoEl.srcObject = localStream;
        // localVideoEl.play();
    }).catch(err => {
      console.error(`getUserMedia exception: err is ${err}`);
    });

The js logging is something like this

The native logging is something like this

2021-01-26 21:07:42.865961+0800 AgoraNgExample[43511:2623107] iosrtc:videoElementsHandler observeVideo() +8s
2021-01-26 21:07:42.877371+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer new() | [element:"[object HTMLVideoElement]"] +0ms
2021-01-26 21:07:42.878016+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#new_MediaStreamRenderer()
2021-01-26 21:07:42.878237+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#init()
2021-01-26 21:07:43.046906+0800 AgoraNgExample[43511:2623107] Metal GPU Frame Capture Enabled
2021-01-26 21:07:43.065696+0800 AgoraNgExample[43511:2623107] Metal API Validation Enabled
2021-01-26 21:07:43.243196+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#run()
2021-01-26 21:07:43.243345+0800 AgoraNgExample[43511:2623107] THREAD WARNING: ['iosrtcPlugin'] took '365.367920' ms. Plugin should use a background thread.
2021-01-26 21:07:43.244000+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer refresh() +2ms
2021-01-26 21:07:43.244168+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer refresh() | no video track yet +1ms
2021-01-26 21:07:43.244347+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer refresh() | [data:{"elementLeft":0,"elementTop":1484,"elementWidth":480,"elementHeight":360,"videoViewWidth":480,"videoViewHeight":360,"visible":true,"opacity":1,"zIndex":0,"mirrored":false,"objectFit":"contain","clip":true,"borderRadius":0}] +1ms
2021-01-26 21:07:43.244657+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#MediaStreamRenderer_refresh()
2021-01-26 21:07:43.244853+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#refresh() [elementLeft:0.0, elementTop:1484.0, elementWidth:480.0, elementHeight:360.0, videoViewWidth:480.0, videoViewHeight:360.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2021-01-26 21:07:43.245134+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer render() [stream:{"_listeners":{"addtrack":[null],"removetrack":[null]},"_id":"1254545C-77FA-4123-B9D2-3F96F203DA91","_active":true,"connected":true,"_audioTracks":{"6C6A7F2F-0F19-4FFA-8FAF-ACD53CD39060":{"_listeners":{"ended":[null]},"id":"6C6A7F2F-0F19-4FFA-8FAF-ACD53CD39060","kind":"audio","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_videoTracks":{"C43BBFBE-6181-4BA2-AFA2-611D73CCB972":{"_listeners":{"ended":[null]},"id":"C43BBFBE-6181-4BA2-AFA2-611D73CCB972","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_blobId":"MediaStream_1254545C-77FA-4123-B9D2-3F96F203DA91","TestListenEvents":true}] +1ms
2021-01-26 21:07:43.245302+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#MediaStreamRenderer_render()
2021-01-26 21:07:43.245385+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#render()
2021-01-26 21:07:43.245786+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer close() +1ms
2021-01-26 21:07:43.245944+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#MediaStreamRenderer_close()
2021-01-26 21:07:43.246016+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#close()
2021-01-26 21:07:43.246071+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#reset()
2021-01-26 21:07:43.246513+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer new() | [element:"[object HTMLVideoElement]"] +1ms
2021-01-26 21:07:43.246620+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#new_MediaStreamRenderer()
2021-01-26 21:07:43.246668+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#init()
2021-01-26 21:07:43.249860+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#run()
2021-01-26 21:07:43.250018+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer refresh() +0ms
2021-01-26 21:07:43.250137+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer refresh() | no video track yet +0ms
2021-01-26 21:07:43.250273+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer refresh() | [data:{"elementLeft":0,"elementTop":1484,"elementWidth":480,"elementHeight":360,"videoViewWidth":480,"videoViewHeight":360,"visible":true,"opacity":1,"zIndex":0,"mirrored":false,"objectFit":"contain","clip":true,"borderRadius":0}] +1ms
2021-01-26 21:07:43.250432+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#MediaStreamRenderer_refresh()
2021-01-26 21:07:43.250529+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#refresh() [elementLeft:0.0, elementTop:1484.0, elementWidth:480.0, elementHeight:360.0, videoViewWidth:480.0, videoViewHeight:360.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2021-01-26 21:07:43.250686+0800 AgoraNgExample[43511:2623107] iosrtc:MediaStreamRenderer render() [stream:{"_listeners":{"addtrack":[null],"removetrack":[null],"update":[null],"inactive":[null]},"_id":"1254545C-77FA-4123-B9D2-3F96F203DA91","_active":true,"connected":true,"_audioTracks":{"6C6A7F2F-0F19-4FFA-8FAF-ACD53CD39060":{"_listeners":{"ended":[null]},"id":"6C6A7F2F-0F19-4FFA-8FAF-ACD53CD39060","kind":"audio","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_videoTracks":{"C43BBFBE-6181-4BA2-AFA2-611D73CCB972":{"_listeners":{"ended":[null]},"id":"C43BBFBE-6181-4BA2-AFA2-611D73CCB972","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_blobId":"MediaStream_1254545C-77FA-4123-B9D2-3F96F203DA91","TestListenEvents":true}] +0ms
2021-01-26 21:07:43.250963+0800 AgoraNgExample[43511:2623107] iosrtcPlugin#MediaStreamRenderer_render()
2021-01-26 21:07:43.251016+0800 AgoraNgExample[43511:2623107] PluginMediaStreamRenderer#render()
2021-01-26 21:07:43.252872+0800 AgoraNgExample[43511:2623107] Failed to bind EAGLDrawable: <CAEAGLLayer: 0x28031c100> to GL_RENDERBUFFER 1
2021-01-26 21:07:43.253017+0800 AgoraNgExample[43511:2623107] Failed to make complete framebuffer object 8cd6
2021-01-26 21:07:43.254657+0800 AgoraNgExample[43511:2623107] Failed to bind EAGLDrawable: <CAEAGLLayer: 0x280315c60> to GL_RENDERBUFFER 1
2021-01-26 21:07:43.254718+0800 AgoraNgExample[43511:2623107] Failed to make complete framebuffer object 8cd6

I also run cordova-plugin-iosrtc-sample with index-local.js in the same enviroment(iphone, xcode, cordova version, plugin version …etc), it can play video stream. I tried to compare logging but they seems very similar, seems they both set videoElement to Observer in js and new a MediaStreamRenderer in swift. I think it should be my usage problem but no idea where to deep into, could you please give me some suggestion? Thanks very much.

Best Regards.

Cordova version: 10.0.0

Cordova iOS version: 6.1.1

Plugin version: 6.0.14

iOS version: 14.3

Xcode version: 12.3

WebRTC-adapter version: latest

WebRTC Framework version: m69

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (8 by maintainers)

Most upvoted comments

At last… I minimized my html code to leave only necessary tags, the video play works. I will check my html bugs tomorrow. I also fallback plugin to 6.0.14 and video play can’t work with getUserMedia throw an exception I think this may be the fixed issue in 6.0.17, as expected. Thx for u guys! @hthetiot @MusabBasheer

octopusjames on Jan 27, 2021

I also fallback plugin to 6.0.14 and video play can’t work with getUserMedia throw an exception I think this may be the fixed issue in 6.0.17, as expected.

That correct, if you use ios 14.3+ you need 6.0.17+, that due the webrtc-adapter overiding iosrtc even if the SHIM is apply after the adapter.

hthetiot on Jan 27, 2021

ahah well done @octopusjames Happy RTC in China 😃

Thank you for closing, and solving by yourself,.

hthetiot on Jan 27, 2021

@octopusjames please view this link https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/docs/videoCSS.md I think you should remove “_src” and the background attribute from the style in video and make the body background transparent.

@MusabBasheer Seems still can’t work, I may try something else. But thank you all the same.

octopusjames on Jan 27, 2021

@octopusjames I think your issue related to video place please check if z-index =-1 specified in video element or via (scss, css) if specified the video will be placed below the wkwebview so it will be no visible.

Good catch @MusabBasheer

Thx for helping.

hthetiot on Jan 27, 2021

@octopusjames I think your issue related to video place please check if z-index =-1 specified in video element or via (scss, css) if specified the video will be placed below the wkwebview so it will be not visible.

MusabBasheer on Jan 27, 2021