cordova-plugin-iosrtc: MediaStream.addTrack does not accept CanvasCaptureMediaStreamTrack
Expected behavior
MediaStream.addTrack should allow to use as a first argument an instance of CanvasCaptureMediaStreamTrack captured from <canvas> by captureStream method.
Observerd behavior
MediaStream.addTrack throws an exception “argument must be an instance of MediaStreamTrack” if the first argument is an instance of CanvasCaptureMediaStreamTrack.
In Chrome/Firefox/Safari an instance of CanvasCaptureMediaStreamTrack inherits from MediaStreamTrack, so it can be used in MediaStream.addTrack method. In cordova app using iosrtc plugin CanvasCaptureMediaStreamTrack is not an instance of MediaStreamTrack, so we have an aforementioned exception in MediaStream.addTrack:
Steps to reproduce the problem
// capture local camera and microphone
navigator.mediaDevices.getUserMedia({ audio: true, video: true }).then(function(stream) {
// get canvas stream
var canvas = document.createElement('canvas');
var canvasStream = canvas.captureStream();
var track = canvasStream.getVideoTracks()[0];
// add canvas video track to the stream
try {
stream.addTrack(track);
}
catch(err) {
console.error(err); // output: Error: argument must be an instance of MediaStreamTrack
console.log(track instanceof MediaStreamTrack); // output: false
}
});
Platform information
- Cordova version: cli-9.0.0 (iOS 5.0.1)
- Plugin version: 6.0.10
- iOS version: 13.3.1, 13.4.1
- Xcode version: 10.2
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 25 (18 by maintainers)
@hthetiot , Please ping me if need my help.
In any case, even if WKWebview was allowing Canvas.captureStream that I’m pretty sure it does not, It will require enormous work to enable converting a “Native” MediaStream to an iosRTC WebRTC MediaStream. But again I already think this is irrelevant since CaptureStream is not even working in Safari therefor it will not work in WKWebview.
This is a free software, I will not implement such features for your blue eyes 😉
This is good way and I will prepare a PR for it.
@hthetiot
Thank you for the detailed explanation about iosRTC MediaStream.
The first part was about
captureStream, which you asked for. The second one - aboutaddTrack.Yes, I do now. But it wasn’t clear at the first time, that I can’t send canvas stream to peer connection. It was looking like I can’t send it to
addTrackbecause it’s not aninstanceof MediaStreamTrack.Sorry, I don’t understand, what do you mean “in term of failure”? I can explain the expected behavior when it’s succeeded, because I already have this behavior in other browsers. You just send
CanvasCaptureMediaStreamTracktopc.addTrackorpc.replaceTrack, and then it just appears (even without renegotiation in case ofreplaceTrack) on the other side of the call as a separate video track which you can attach to any<video>.Yes. What exactly is not logic - having more than one video tracks in one media stream which then will be sent over one peer connection? For example, if I want to send camera track (in small resolution, to show in the upper left corner of the recepient’s screen, with the ability for him to hide the camera separately from other video tracks) + screen capture track (in maximum resolution, to show in full screen) + canvas capture track (which can be shown separately from camera and screen) - it does not make any sence?
I proposed a working WebRTC scenario from other browsers, before you explained that it’s not possible in iosRTC because
MediaStreamTrackprovided by iosRTC is not even close to nativeMediaStreamTrack. I didn’t develop iosRTC, so I couldn’t know this.No. I expected to combine two or more video tracks from different sources (camera, screen, canvas) in one media stream and then send it via one peer connection.
Yes, thank you for the example.
Now, it’s totally clear, after you explained that it’s a way deeply than just a type incompatibility issue.
This is exactly what I expected and why I opened this issue.
Yes, of course.
Yes, but it is not the reason to not deliver new features to up-to-dated users, especially if the market share of iOS 13+ is 93% (98% for iOS 11+, where
captureStreamwas appeared the first time in Sep 2017). And you can always show something like “This feature is available only on iOS 11+”.No, I don’t, why would I? I want two separate and compressed video tracks - one for the screen and another one for the witeboard, which will eat a minimum amount of traffic because it will be static almost all the time. I’m not saying this is the best way to implement a whiteboard. I’m just saying this way is good enough to live, at least from the first look.
Yes, this is probably the best way in terms of traffic and performance, thank you. But it’s a way more complicated than just adding additional video (canvas) track into the already established peer connection and then just attach it to
<video>on the other peer. But, of course, if you need a two-way whiteboard with the ability to undo the other peer’s actions, for example, then yes, the data channel is probably the best way.Thank you.
@hthetiot
I’m not asking you to implement any features for me. I’ve already managed this without intermediate local stream. I’m just reporting an incompatibility issue in
MediaStream.addTrackmethod.You did a lot of work to make this plugin compatible with W3C WebRTC standard. I thought reporting this issue would help in this. Also, CanvasCaptureMediaStreamTrack is a good way to implement collaboration features like whiteboard, bringing new user experience in cordova apps, which is also good.