PowerShell: 7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows
Prerequisites
- Write a descriptive title.
- Make sure you are able to repro it on the latest released version
- Search the existing issues.
- Refer to the FAQ.
- Refer to Differences between Windows PowerShell 5.1 and PowerShell.
Steps to reproduce
With versions prior to 7.4.0, pressing Tab to get completion on a path with ~
converted that path into the full path for my home folder.
Type dir ~\down
and press Tab:
- In 7.3.9, it will be completed as
dir C:\Users\bradwilson\Downloads\
- In 7.4.0, it will be completed as
dir ~\Downloads\
For built-in commands (like dir
), this is fine. For executables, this passes a path with ~
in it, which makes using such paths impossible unless the executable has its own support for mapping ~
to the home folder. For example, if I try to run notepad ~\.config\git\config
, it will tell me the path isn’t found; if I run notepad C:\Users\bradwilson\.config\git\config
, it will open the file appropriately.
This breaks more than a decade of muscle memory expecting ~
to be translated into the correct path, as I’ve been using PowerShell as my shell since it was called Monad.
This appears to have been purposefully introduced in #19489. I cannot find any way to restore the old behavior short of sticking w/ version 7.3.9.
Expected behavior
PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches
CompletionText ListItemText ResultType ToolTip
-------------- ------------ ---------- -------
C:\Users\bradwilson\Downloads Downloads ProviderContainer C:\Users\bradwilson\Downloads
Actual behavior
PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches
CompletionText ListItemText ResultType ToolTip
-------------- ------------ ---------- -------
~\Downloads Downloads ProviderContainer C:\Users\bradwilson\Downloads
Error details
No response
Environment data
Name Value
---- -----
PSVersion 7.3.9
PSEdition Core
GitCommitId 7.3.9
OS Microsoft Windows 10.0.22631
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Name Value
---- -----
PSVersion 7.4.0
PSEdition Core
GitCommitId 7.4.0
OS Microsoft Windows 10.0.22631
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Visuals
No response
About this issue
- Original URL
- State: open
- Created 7 months ago
- Reactions: 6
- Comments: 67 (16 by maintainers)
@kilasuit:
~
part of the PR (with the variable-preservation part yet to come) is: #5350@237dmitry, thank you for your clarification that there may be a language barrier and for your expression of respect.
Just to recap the previous clarification: it isn’t utilities (external programs) that interpret
~
, it is the (POSIX-compatible) shell that up front resolves~
to the value of$HOME
and passes the result to utilities.PowerShell (sensibly) chose to emulate this behavior - selectively, for calls to utilities (external programs) only - but currently only on Unix-like platforms. It could equally choose to do that on Windows too, which would solve the issue at hand. Again, it comes down to whether we think this would break anything.
To resume the framing debate:
It is clear that a change must be made.
Reverting to the old behavior is one option - possibly selectively on Windows.
Let me provide some context:
On Unix-like platforms it is the (POSIX-compatible) shell (such as
bash
) that performs tilde (~
) expansion, not the standard utilities (such as asls
) themselves; the same applies to globbing (wildcard expansion in filenames), among other such so-called shell expansions. You can verify this with/bin/ls --% ~
from PowerShell.On Unix-like platforms only, PowerShell emulates this behavior (as well as globbing) when calling external programs:
~
- either in isolation of if followed by/
(but not\
) - to the value of$HOME
.`
to escape~
-/bin/echo `~
- should work like/bin/echo '~'
and/bin/echo "~"
, i.e. should suppress expansion, but currently doesn’t; the inverse is true for/bin/echo ~/'foo bar'
, which should expand -d due to the~
itself being unquoted, but doesn’t (if you remove the space, it works): #20754~user1
works in POSIX-compatible shells to refer to the home directory ofuser1
(verify from PowerShell withsh -c "echo ~$env:USER"
), but not in PowerShell: #12387On Windows, where the native shell (
cmd.exe
) has no such expansion features, PowerShell does not perform them.Given the above, I propose the following variation of @MartinGC94’s proposal 1:
~
expand to the value of$HOME
only there - conceivably, it could even expand to verbatim$HOME
, so as to preserve the intent to express the path abstractly.While making PowerShell emulate literal
~
expansion on argument-passing to external programs on Windows also may seem appealing, there’s a greater risk of breaking existing Windows-only code that never anticipated~
as having special meaning and therefore requiring quoting. Also, emulating globbing too is definitely not an option, as a program expecting, say,file*.txt
verbatim, then receiving multiple arguments (file1.txt file2.txt ...
) could break.Yes, as you’ve discovered this was an intentional change and there’s currently no way to change the behavior. A quick workaround you can implement on your own machine is to update the tabexpansion2 function to replace the
~
character in the completion results with your home path.Assuming this is something that people want fixed, I can think of the following solutions:
~
before passing it in as an argument to native commands.Quick and dirty tabexpansion2 workaround:
Let me try another, hopefully comprehensive summary that may serve as the basis for deciding how to resolve the problem at hand:
Context:
The problem at hand arose from an intentional change with unintentional consequences:
~
in the completed argument instead of the previous behavior of instantly expanding to the literal, full path of$HOME
.~
paths, because the current lack of tilde expansion during parameter binding on Windows passes~
verbatim, resulting in native programs seeing a non-existent path.~
to the full, literal path of$HOME
) is performed by PowerShell, as a nod to the behavior of the platform-native POSIX-compatible shells, so the change in tab-completion behavior has no ill effects there (with the exception of #20851, but that’s a separate bug).Solution options:
(a) Revert to the old behavior - at least for now - and possibly only on Windows.
(b) Again possibly only on Windows, replace
~
with$HOME
(only when calling native programs) to preserve the spirit of #5350 without breaking anything.© Bring PowerShell’s tilde expansion to Windows too, as @domsleee’s PR #20402 attempts.
It is currently planned as an experimental feature, which means that not only would it take too long to become a stable feature, but isn’t even guaranteed to become one.
However, if a consensus can be reached that this change should be made a stable feature right away, it would solve the problem at hand in a timely manner.
~
in isolation or start with unquoted~\
or~/
and that the target program did not mean to expand the~
to the value of$HOME
itself anyway.cmd /c echo ~\foo
currently echoes verbatim~\foo
but would then echo, say, verbatimC:\Users\jdoe\foo
.(Potentially) separate considerations and workarounds:
Set-CompletionOptions
cmdlet from @MartinGC94’s PR #19518.TabExpansion2
function, as shown in @MartinGC94’s comment.I’m surprised by the number of people who are suggesting that I’m asking for a new feature rather than lamenting of the removal of an old feature. The conversations about “confusions for users” is exactly why we’re here: a “breaking” change was made and me (a user) is confused about why it was done and how I can undo it. This was clearly a feature I counted on and would like to have back, even if it is no longer the default behavior.
I would be fine with tab completion of
~
into$HOME
(under the assumption that it would be evaluated before being sent to executables). It’s also worth noting that in both 7.3.9 and 7.4.0 if I typedir $HOME/Down
and press Tab it gets expanded toC:\Users\bradwilson\Downloads\
or/home/bradwilson/Downloads/
(depending on OS).(I just deleted a bunch of stuff because I realized that
echo
in my pwsh examples was using the echo alias and not /usr/bin/echo, so my examples were incorrect.)This marks the second time in this discussion that the behavior of Linux shells has been misdescribed. To be clear, Linux programs DO NOT interpret ~/path. Period. They never even SEE the
~
part. The shell itself replaces ~ with a reference to the user’s home directory, and it is the lack of this behavior on Windows that has rendered practically every utility I use non-functional in the past several days.This is a serious breaking change, and the damage needs to be undone asap. If I enter a path beginning with ~ and press tab, I expect to see a real path. Period.
Edit: as far as HOW the damage is undone, I don’t care. I primarily use Linux. If you make it work like Linux, that’s fine. If you make it work like Windows, that is also fine. The problem right now is that it doesn’t work at all. Whatever you do, however, do not pretend it’s reasonable to expect me to type
$HOME
instead of~
. One of them is a little longer than the other, you see. 😃Consider this case, from the completion where I’m trying to open
~/my folder/a.txt
, usingcode ~/my<tab>
(on mac)Of course it shouldn’t, you quoted the tilde. Quoting tilde prevents expansion in
bash
Which was fine, because you could rely on tab completion of ~ to get a full path for the executable. Hence why I opened this issue.
That evaluates the variable before launching the executable with the resulting string
That leaves ~ as a meta character which the executable may or may not understand, if you’re calling notepad this is the same doing
notepad ~/Documents/mydoc.txt
from the Run box , or in cmd.Transforms ~ to home when you hit enter. (With a simple check to see if it looks like the start of a path)
I came here after some googling. This should be reported in the release notes What’s New in PowerShell 7.4
To summarize: this works fine:
But this doesn’t:
Maybe there is a way to treat
~
like$HOME
But @mklement0 already said all that in his complete comment.
For what it’s worth, the problem I have is that I don’t think you went far enough. I really don’t want TabCompletion to replace generic values like
~
with the specific value on my computer. The same goes for or$home
and$pwd
and any other variable that I might have a partial path in.How about instead of reverting a good feature, we replace the literal
~
at the front of the path with$HOME
and then … leave all the variables as variables in the path?Since PowerShell expands variables when passing arguments to native apps, this gets us the best of all worlds:
~/Doc{Tab}
still tab completesNOTE: the catch to this is that it doesn’t work if they’re single-quoting the path they are tab-completing, in which case we might want to resort to the old behavior…
And you are welcome to put this
PSReadLineWithAutoCorrection
as a module in thePSGallery
, but it does not need to be in the core project.$HOME is interpreted as home directory on all PowerShell platforms. As $Env:USERPROFILE on Windows and $Env:HOME on UNIX. So if you want to write portable scripts, use $HOME.
I don’t want autocorrect ever on a command prompt that changes the command from what I think I am executing to something different without me confirming that is what I want. I don’t care if it is Clippy, AI, ChatGPT, Code Pilot or Mechanical Turks. I want to execute the commands that I type and get the appropriate error if its wrong. If I want to cycle through the options then I can use the tab-expansion.
… and I want the interpretation of what I type to be exactly the same as when I have the same text in a powershell script.
@mklement0, thanks for raising that issue.
@jhoneill, is your proposal to make the “enter autocorrect” a built-in option is
PSReadLine
? I would much prefer tilde expansion described in #20402, since it would allow tilde expansion in scripts. A lot of bash scripts use it, so it would be nice for linux users coming from bash as a default in a future release of powershell (assuming all goes well and there isn’t some legacy windows code that relies on the tilde not expanding).If “enter autocomplete” was implemented as an option in
PSReadLine
, I wouldn’t expect it to be enabled by default, since it would surprise users when their prompt changed after pressing enter (I believe @rhubarb-geek-nz was saying this).Question - Would the enter autocorrect be useful if #20402 was implemented? Any advantages compared to #20402?
And if it were invisible this would be a problem, but what happens is the text between the prompt and the cursor changes
I hit enter and
Well, you can’t see run
notepad ~/myfile
to find out why it doesn’t work because every time you try the line changes before your eyes, giving a you a strong hint to change what is in the script. And because this is a personal choice for users who want it, if they regularly use ~ in scripts (which I’m saying “don’t do”) they need to take responsibility for handling the difference.No it’s an autocorrect. imagine if some of the light bulbs in VS code just changed
sort
toSort-object
(to deal with sort not being an alias on linux and running the external executable), or there was one for ~/ paths which didn’t say “don’t use this” but changed it. I really don’t see it as any different to word converting(c)
to a copyright symbol. Bottom line its there if people want it as a workaround, but it’s not the universal answer.The problem with ~ being invisibly interpreted as $HOME in PSReadLine but not when executing a script leads to the problem where you can’t see why your script doesn’t work. You type the line at the command prompt and it works, you run the script and it doesn’t work even though the text is identical. So in effect you would have two different interpreters with different rules both claiming to be PowerShell.
@domsleee, good find on the broken tab-completion of
~/my
to'~/my folder'
, and thanks for tackling a PR for~
expansion on Windows too (https://github.com/PowerShell/PowerShell/pull/20402).The broken tab-completion is conceptually related to #20754 (the fact that even a properly specified
~/'my folder'
doesn’t shell-expand to the equivalent of$HOME/my folder
); I’ve created a separate issue for it:Yeah, I have no issue with the tilde expansion behaviour on mac, only the tab expansion. The problem is the tab expansion adds the single quotes around the tilde for folders with spaces, so
code ~/my<tab>
turns intocode '~/my folder/'
.To be clear I’m suggesting a reason the new behaviour is undesirable for mac, from https://github.com/PowerShell/PowerShell/issues/20750#issuecomment-1825707096:
pwsh 7.4.0 (mac):
pwsh 7.3.9 (mac):
I would be one of these people 💯 #20402 Although I would only expect tilde expansion to only occur without quotes, so the proposed flag
PSNativePSPathResolution
would not be considered a solution to the tab completion issues described here 👍I think the current completion doesn’t make sense on nix either, because you can’t reuse the completions in native executions. Consider this case, from the completion where I’m trying to open
~/my folder/a.txt
, usingcode ~/my<tab>
(on mac)The old completion from 7.3.9 (
code '/Users/user/my folder/$'
) was reasonable behaviour, 7.4.0 isn’t 👍 I’m not saying reverting is the only way, just saying this is a reason to change to the 7.4.0 unix behaviour as well.You may not be proposing that, but that is the effect. Interactive interpretation would be different to that from a script, so ~ will have one interpretation when typed, and a different one when in a script.
I may be misinformed then. In any event prior to PowerShell, everything on linux seemed to work with ~ and nothing on Windows did. There are / were different expectations on the the different OSes.
Same with / on windows and linux PowerShell try
cd Temp: ; cd /
one moves you to the root of the “temp” drive and one to the root of everything. Making cd / change to the root of Temp: on linux would upset a good number of Linux users who expect it to go to / not temp:\ - but changing to C: on Windows would seem even more perverse.Not quite what I said. We tell people not to write aliases in scripts and ~ is an alias for $Home, so the advice don’t write
"~/Documents"
but"$home/Documents"
seems fine to me. I’m sure someone gave me that advice 10 or 15 years ago 😃I’m not proposing that. I’m saying
PowerShell on Linux has to do the translation. The operating system “exec” call does not expand variables or interpret ~, it is done by the shell.
That would be the worst of all worlds, ~ working in Linux scripts, ~ working in Linux command line, ~ working in Windows command line but not in scripts.
How could you interactively test commands if the evaluation of a command is different depending on whether it was typed or from a script?
I dislike the idea of invisible translation that occurs when in an interactive session that is different from what happens when it is part of a script.
Interpretation of ~ and $HOME should be consistent whether entered in by hand or part of a script.
This script expands the ~
Which then leads to why do we need to invent new rules for the Windows handling of ~, why can’t we use the same rules as on Linux?
Thanks, @bradwilson; note that such issues (pertaining to PowerShell’s incomplete emulation of Unix-style tilde expansion on Unix-like platforms) not only have already been filed, but were previously linked to above.
From what I can tell, this comment still summarizes what’s needed to resolve the issue at hand, and nothing new was added since.
@archer884 I’ve updated the title to say “on Windows” so perhaps the discussion about Linux behavior (right or wrong) can end up in a new issue (if, for example, people want to file a bug against the fact that
~user
doesn’t work correctly).Yes, thanks, but this issue has nothing whatsoever to do with UNIX shells. This issue relates to the interactive user experience on Windows, which is entirely broken as of version 7.4.
I think this dilemma will be successfully resolved without affecting the already running version on the Unix platform. As you noted in another thread, there are problems with quotation marks, but quoting only part of the path is not that common in everyday interactive practice.
@MartinGC94 I think this really needs to be rolled back to previous behaviour. But thanks for sharing the modified
tabexpansion2
I’ve created a gist https://gist.github.com/jhoneill/322a77199350c76a5785f5406ea97bac with what I think is a more effective version although I’m not sure about modifying
$inputScript
and then changing theReplacementlength
after callingCompleteInput()
However it does some things which I had in an argument completer previously and had to be bound to whatever cmdlet/function arguments I thought would benefit - a bit tedious
$userProfile
and tab expands as if I had typedc:\users\james\
or whateverAdminTools, ApplicationData, CDBurning, CommonAdminTools, CommonApplicationData, CommonDesktopDirectory, CommonDocuments, CommonMusic, CommonOemLinks, CommonPictures, CommonProgramFiles, CommonProgramFilesX86, CommonPrograms, CommonStartMenu, CommonStartup, CommonTemplates, CommonVideos, Cookies, Desktop, DesktopDirectory, Favorites, Fonts, History, InternetCache, LocalApplicationData, LocalizedResources, MyComputer, MyDocuments, MyMusic, MyPictures, MyVideos, NetworkShortcuts, Personal, PrinterShortcuts, ProgramFiles, ProgramFilesX86, Programs, Recent, Resources, SendTo, StartMenu, Startup, System, SystemX86, Templates, UserProfile, Windows
MyDocuments, MyMusic, MyPictures, MyVideos
C:\users\james\documents
C:\users\james\documents\p
and cycles through things beginning with P.
after 2 to …\ so cd … [tab] cycles through directories 2 levels up.I saw a way to fix #20765 at the same time so I’ve put that in at the end
In fact, when this behavior finally appeared in 7.4.0-rc.1, I was glad about it. But perhaps this is not as relevant in Windows as it is in Linux.