tauri: [bug] Lots of internal_on_mousemove IPC calls causing lag and cursor flashing in 2.0.0-beta.0

Here is a close up of the flashing cursor I noticed (I am not clicking or anything, just moving the mouse around over the button)

https://github.com/tauri-apps/tauri/assets/25918877/0fba563c-42d2-4f2d-80ec-2970286fe041

I ran into this issue as well and it basically made tauri unusable for me. I wrote up a quick workaround for the odd mouse behavior using the isolation pattern.

When my app starts (svelte in this case) I get the window size and wait for the iframe __tauri_isolation__ to load. Once loaded I send it the initial size and then subscribe to tauri://resize event and send it the updated size onwards.

import { Window } from "@tauri-apps/api/window";
import { onMount } from "svelte";
  
const win = Window.getCurrent();
let iframe: HTMLIFrameElement;

async function updateDimensions() {
    if (!iframe?.contentWindow) return;

    const size = await win.innerSize();

    iframe.contentWindow?.postMessage(
      {
        type: "resize",
        width: size.width,
        height: size.height,
        is_fullscreen: await win.isMaximized(),
      },
      "*"
    );
  }
 
onMount(async () => {
    while (!iframe?.contentWindow) {
      iframe = document.getElementById(
        "__tauri_isolation__"
      ) as HTMLIFrameElement;
      await new Promise((resolve) => setTimeout(resolve, 100));
    }

    await updateDimensions();
});

win.listen("tauri://resize", async () => {
  await updateDimensions();
});

This is what I do in the isolation app:

let width = 0;
let height = 0;
let is_fullscreen = false;

window.addEventListener("message", function (event) {
  // maybe the origin should be checked and some other stuff idk

  if (event.data.type === "resize") {
    width = event.data.width;
    height = event.data.height;
    is_fullscreen = event.data.is_fullscreen;
  }
});

window.__TAURI_ISOLATION_HOOK__ = (payload) => {
  if (payload.cmd === "plugin:window|internal_on_mousemove") {
    let x = payload.payload.x;
    let y = payload.payload.y;

    let min_x = 10;
    let min_y = 30;

    // console.log(`x: ${x}, y: ${y}, width: ${width}, height: ${height}`);

    if (is_fullscreen || !(y < min_y || y > height - min_x || x < min_x || x > width - min_x)) {
      payload.cmd = "nothing";
      payload.payload = {};
    }
  }

  return payload;
};

It still spams ipc calls which isn’t ideal and I’m waiting for a fix but this fixes the stuttering issue and be sure to make a tauri command that doesn’t do anything so that the console isn’t spammed with errors about the endpoint not existing.

Edit: I don’t see why this behavior is needed if the app is decorated so in the code i sent above you can use win.isDecorated() and override the event(s) related to window resizing

Small possible remaining improvements (I might give them a go but should not hold up the current PR):

1. Handle changing cursors entirely in JS, no IPC

   1. Will need to fetch decorated, resizable, maximization status from Rust and know when this changes by evaluating a script on the page.

   2. Get DPI and set border region size in pixels in JS (what if the window moves between monitors or spans two monitors with differing DPI?)

The reason why I didn’t go with this solution before, is that reloading the webview will make it lose the stored state or get out of sync. Maybe a mix of IPC and stored state could work.

1. When within border region, set cursor in JS using CSS targeting html element (User CSS might interfere with this?)

Did test this idea before and it worked pretty well, however we need to use weird class names so as to not conflict with user (i.e. __TAURI_INTERNAL_CLASS_nw-resize__)

1. Try adding resizing borders to the HTML around the window with corresponding CSS for cursors

   1. The user could possibly remove these if they clear out the document body or set a z-index that is higher than these borders, might be too flimsy. Would need to observe these elements and add them back if they are removed.
   2. Attach events directly to these HTML elements if possible.

I don’t agree with this idea at all, since it will interfere with the user DOM hierarchy and could lead to hidden behaviors that they can’t explain.

1. Only attach the mousemove event for sending resizing IPC calls when the left click mousedown event is fired, and remove it after mouseup.

This will prevent the cursor from changing when on the window edge and users won’t know if they are on a resizable edge or not.

tao does support resizing undecorated windows natively, however when adding a wry webview, which is basically WebView2 on Windows, it is not a simple widget that is added to the tao window, it creates a new child HWND that fills the tao window and so any events related to mouse clicks will be on their HWND and won’t reach to the underlying tao window.

The implementation of this is quite hevavy for just to change the cursor style and check able to resize. MouseMove trigger very frequently. maybe need a debounce

Amr created a PR that is awaiting approval that fixes these issues. The OS check and permissions issue has been resolved. The OS check is now performed in Rust and checks for Windows alone, no script is attached for non-Windows platforms. And the permissions issue is avoided by using a custom invoke system (window.ipc.postMessage) based on postMessage/eval that avoids the permissions & runtime authority.

Here is a close up of the flashing cursor I noticed (I am not clicking or anything, just moving the mouse around over the button)

2024-02-06_09-29-46.mp4

Nice catch, I will fix that as well in the PR

For transparency, the solution in #8537 was already used before in wry so the performance should be the same and because webview2 doesn’t provide anyway to hit test the webview from native side, however on Linux, we used to use raw GTK APIs and hit test on the native side, I will see if I can bring that back.

Probably should have included the error in https://github.com/tauri-apps/tauri/issues/8750 lol. but yes, we agree that it’s too hard on the ipc and will look into it. (that said, i don’t see the cursor flashing)

Edit: I closed my own issue cause yours is better, for visibility here’s my comment from the other issue:

This is mostly just a reminder issue because i have to log off for today.

The resizing implementation we introduced in https://github.com/tauri-apps/tauri/pull/8537 is pretty hard on the ipc, even if the window uses default decorations. I only noticed this because i was missing the default window permissions so i had thousands of error messages in the devtools console complaining about a forbidden internal command (which tbh is also weird).

Ideally the mousemove event could be handled in rust only but i’d imagine there’s a reason we didn’t do that so maybe the best we can do is to disable the listener in undecorated windows?