terminal: Incorrect content copied from terminal (Japanese Toothpaste?)

Windows Terminal version

1.17.1023

Windows build number

10.0.19044.0

Other Software

tmux 3.0a

$ cat ~/.tmux.conf| without_comments
set -g default-shell /usr/bin/zsh
bind S source-file ~/.tmux/bastion-quad
set -g default-terminal "tmux-256color"
set -g history-file ~/.tmux_history
set -g history-limit 150000
set-environment -g 'IGNOREEOF' 10
set -g prefix `
unbind -n C-l
unbind -n C-t
unbind -n C-u
unbind -n C-y
unbind -n C-d
unbind -n C-b
unbind C-d
bind ` send-prefix
bind r source-file ~/.tmux.conf \; display "Tmux config reloaded"
bind c new-window -c "#{pane_current_path}"
bind - split-window -v -c "#{pane_current_path}"
bind = split-window -h -c "#{pane_current_path}"
bind -n C-S-Left swap-window -t -1\; select-window -t -1
bind -n C-S-Right swap-window -t +1\; select-window -t +1
bind -n C-o select-pane -t :.+
bind H resize-pane -L 5
bind J resize-pane -D 5
bind K resize-pane -U 5
bind L resize-pane -R 5
bind-key X set-window-option synchronize-panes\; display-message "synchronize-panes is now #{?pane_synchronized,on,off}"
setw -g mouse on
setw -g automatic-rename off
set-window-option -g mode-style bg=colour25,fg=colour254
set -g pane-border-style fg=colour28
set -g pane-active-border-style bg=default,fg=colour98
set -g status-justify centre       # center window list for clarity
set-option -g status-position top  # position the status bar at top of screen
setw -g monitor-activity off
set -g visual-activity off
set -g status-style bg=colour28,fg=default
set -g window-status-style fg=colour7
set -g window-status-current-style fg=colour1,bg=colour7
bind m run "tmux new -d -s second; tmux movew -t second;"
bind M run "tmux movew -t base;"
set -g status-interval 300
set -g status-right-length 60
set -g status-right "#[fg=colour73] %d %b %Y %l:%M %p"
set -g @override_copy_command 'clip.exe'
set -g @plugin 'tmux-plugins/tpm'
set -g @plugin 'tmux-plugins/tmux-yank'
run '~/.tmux/plugins/tpm/tpm'

Here’s terminal’s interaction settings image

Steps to reproduce

This seems to only be an issue in TMUX… however I’ve tested this in the current stable terminal version of 1.16.10261.0 and it is not a problem

try to copy the text sl iam token refresh. You have to be careful not to get any pre or post characters.

  • as far as I can tell, the terminal colouring has no effect on this

empty clipboard image

selecting text to copy image

Japanese text copied to clipboard image

selecting text to copy - with additional text at beginning image

proper text copied to clipboard image

Expected Behavior

copy text sl iam token refresh into clipboard

Actual Behavior

copies out text ę±³ę¤ ęµ”ē ę­Æę¹„ēˆ ę™„ę•²ę”³ which when pasted into google translate, is apparently Japanese for toothpaste

image

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 24 (11 by maintainers)

Most upvoted comments

so I’m guessing windows terminal understands the OSC 52 escape sequences for clipboard setting?

Absolutely šŸ˜„

Okay, so I’ve got some good news and some bad news.

This is one of the amazing strings that triggers a Windows function called IsTextUnicode to go haywire. Check this out.

is_unicode.cpp

#include <stdio.h>
#include <Windows.h>
int main() {
        const char* foo = "sl iam token refresh";
        BOOL ret = IsTextUnicode(foo, 20, NULL);
        fprintf(stderr, "Ret %d\r\n", ret);
        return 0;
}

output

$ ./is_unicode.exe
Ret 1

So, the thing about IsTextUnicode is that it makes guesses. Literally! The text is otherwise totally unencoded, so it looks like it might actually be Unicode.

clip puts it on the clipboard as a Unicode string…

Now, if we manually convert sl iam token refresh from UTF-16 to UTF-8, we get…

$ iconv -f UTF-16LE -t UTF-8 < /mnt/c/Users/duhowett/foo.txt
ę±³ę¤ ęµ”ē ę­Æę¹„ēˆ ę™„ę•²ę”³

toothpaste


This all tickled a memory. If you’re interested in following along with me, it looks like Notepad used to have the same problem. Here’s why:

this is by far the best issue on our repo

Nevermind, I can totally repro this. What on earth? Maybe we can fix clip…

hahaha, it’s the ghost of clippy šŸ“Ž coming back to haunt us!

What on earth…! This is amazing!

It looks like you’re using tmux’ copy mode. Would you be able to grab us a trace of what’s coming out of tmux by using the Debug Tap?