electron: sendInputEvent doesn't work for iframe/webview

I’m trying to interact with an embedded video using manual mouse input via webContents.sendInputEvent api. It stopped working in 5-0-x.

Issue Details

Electron Version:
- 5-0-x, 6-0-x
Operating System:
- Win10 (1903), macOS 10.14.6
Last Known Working Electron version:
- 4.2.10

Expected Behavior

sendInputEvent should produce the same effects as OS input

Actual Behavior

sendInputEvent doesn’t interact with <iframe> or <webview>

To Reproduce

const { app, BrowserWindow } = require('electron')
// Fails with 5.0.10
// Works with 4.2.10

app.once('ready', () => {
  const { webContents: wc } = new BrowserWindow({
    width: 800,
    height: 600
  })
  wc.once('did-finish-load', () => {
    setTimeout(() => {
      wc.sendInputEvent({ type: 'mouseMove', x: 600, y: 300 })
    }, 1000)
    setTimeout(() => {
      wc.sendInputEvent({ type: 'mouseDown', x: 600, y: 300, button: 'right' })
    }, 2000)
    setTimeout(() => {
      wc.sendInputEvent({ type: 'mouseUp', x: 600, y: 300, button: 'right' })
    }, 2100)
  })
  wc.on('context-menu', (event, params) => {
    console.log(params)
  })
  wc.loadURL('https://www.w3schools.com/html/tryit.asp?filename=tryhtml_youtubeiframe')
})

Additional Information

The test above manually triggers a right click on the embedded video, which should

activate iframe for the first mouse move
open the youtube context menu instead of main frame’s

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 1
Comments: 23 (3 by maintainers)

Commits related to this issue

Make keyboard events propagate to iframes ref: https://github.com/electron/electron/issues/20333#issuecomment-730335643 — committed to c0re100/electron by c0re100 a year ago
Make keyboard events propagate to iframes ref: https://github.com/electron/electron/issues/20333#issuecomment-730335643 — committed to c0re100/electron by c0re100 a year ago
Make keyboard events propagate to iframes ref: https://github.com/electron/electron/issues/20333#issuecomment-730335643 — committed to c0re100/electron by c0re100 a year ago
Make mouse events propagate to iframes/webviews ref: https://github.com/electron/electron/issues/20333#issuecomment-723056236 — committed to c0re100/electron by c0re100 a year ago

Most upvoted comments

Bump.

george-thomas-hill on Jun 7, 2023

Still an issue.

Jelmerro on Apr 12, 2023

This is still very much relevant, please consider implementing this, it would make so much more sense to not have to worry about which frame is where and send the event to the right frame (which barely works, as it’s not a user event in that case 😢).

Jelmerro on Oct 10, 2022

It seems to be working for normal case if I patch the content::RenderWidgetHostImpl::ForwardMouseEvent method to this:

void RenderWidgetHostImpl::ForwardMouseEvent(const WebMouseEvent& mouse_event) {
  ...
  // NOTE: (Modified) Route mouse event just like in RenderWidgetHostViewEventHandler::OnMouseEvent
  MouseEventWithLatencyInfo mouse_with_latency(mouse_event, ui::LatencyInfo(ui::SourceEventType::MOUSE));
  delegate_->GetInputEventRouter()->RouteMouseEvent(
		  GetView(), &mouse_with_latency.event, mouse_with_latency.latency);

  /*ForwardMouseEventWithLatencyInfo(mouse_event,
                                   ui::LatencyInfo(ui::SourceEventType::MOUSE));*/
  ...
}

However, I don’t know how does this change fit the overall chromium structure or is okay at all.

Also, I want it to work with OSR primarily, and that still fails to work even with these changes

UPDATE: I managed to solve the OSR case as well with modifying contents::RenderWidgetTargeter::FindTargetAndDispatch.

If we call QueryClient unconditionally, then everything seems to be fine. Although in this case we actually don’t use the viz hittester code path, so I’m kinda lost here why it’s there or OSR should use it or not. 😕

pergy on Sep 27, 2019