vstest: Runsettings from command line (--) doesnt have capability to pass TestRunParameters

please reopen this. This is crucial for our integration with Azure Dev ops

Closing; reactivate if still required.

It seems like this is still unresolved - why was it closed so hastily? The thing that makes the least sense is how it works right now, which is to simply throw up a random XML error

I also need to be able to modify TestRunParameters from cmdln. Please re-open this issue.

A little ugly but you could do something along the lines of: dotnet test -- TestRunParameters/Parameter[contains(@name,'SETTINGNAME')]/@value=SETTINGVALUE

Left-hand-side of the ‘=’ can be xpath that needs to resolve to an existing attribute or element.

Note: that the xpath cannot contain ‘=’.

Yes pleaaase

This still doesn’t seem to be possible. Can this issue be reopened?

Please make this work!

So is this possible now? 😐

There is no way to set TestRunParameters from the command line using – RunSettings. This issue needs should never have been closed as that was the original topic. All other discussions are work arounds to the fundamental issue of needing to set run parameters dynamically during the CI process without needing a runsettings.xml file to manipulate.

Not having this capability makes it difficult to use Azure DevOps “secret” pipeline variables from unit tests. Secret variables are not automatically added to environment variables, and not having this feature means there’s no way to explicitly reference the pipeline variable from the .NET Core CLI task and pass it in on the command line.

Yay reopened 😃

Still no way of passing cmdline to the runsettings? Seems a bit silly to close the issue without any resolution.

Why was this issue closed without a resolution? I’m also facing this with vstest.console.exe`. My scenario is to override some property already defined in my runsettings file through commandline arguments.

It might be better to add testrunparameters as first class parameter in the cmdline.

dotnet test --TestRunParameter:webAppUrl="http://localhost --TestRunParameter:webAppUser=faizan

This was fixed in the master of dotnet/sdk, so it will be part of the next 5.0 release. Checking with the team to get the simplest way to get it into 3.1 as well.

I could not get it to not require any escaping, that would only be possible on Windows via PInvoke, and even that was not 100%. So instead I aimed to make it the same syntax for dotnet test, dotnet vstest and vstest.console. It needs a single escapement, and will merge your test run parameters if they contain spaces. Typically ", value or when you provide value with spaces. In powershell --% parsing terminator is the secret sauce. On bash you need to escape all spaces quotes and parentheses, but the error messages are helpful. It also aligns nicely with the current error vstest console message if you do it wrong.

# cmd
dotnet test  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

# powershell
dotnet test --%  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\") 

# bash
dotnet test -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\) 

At the moment I am waiting for PR to dotnet/sdk to build to see if I pass their tests. It works locally on my Windows.

@qidydl I tried several version but cannot figure out how to do get this working. Here’s what I tried:

dotnet test -- 'TestRunParameters.Parameter(name="myParam", value="myValue")' dotnet test -- "TestRunParameters.Parameter(name=\"myParam\", value=\"myValue\")" dotnet test -- "TestRunParameters.Parameter(name=\\\"myParam\\\", value=\\\"myValue\\\")"

Now I always get: One or more runsettings provided contain invalid token

Any ideas?

BTW, would be nice if it could tell me which token is invalid.

@nohwnd I agree, does seem like they are being a bit too strict. I think it is literally a one char change (w to S) to get this to align with what Azure DevOps currently supports.

To be clear, currently this works: dotnet vstest tests.dll -- TestRunParameters.Parameter(name=\"MainGroupBrowser\",value=\"Chrome\") But this does not: dotnet vstest tests.dll -- TestRunParameters.Parameter(name=\"MainGroup:Browser\",value=\"Chrome\")

@mikanyg thanks for reporting back, I saw it as well in the issue that is linked just above your comment, see my original comment below:

@TroyWalshProf will close your issue #2377, and will track the improvements here.

_originally posted by @nohwnd in https://github.com/microsoft/vstest/issues/2377_ Please see https://github.com/microsoft/vstest/issues/862#issuecomment-603894275 for some workarounds and explanation. You will need to do some escaping, which will also depend on whether you run the commands from cmd or powershell. Because vstest does not route via msbuild you should be able to escape it like this, and in the same way for vstest.console.

# powershell
dotnet vstest tests.dll -- 'TestRunParameters.Parameter(name=\"Browser\",value=\"Chrome\")'

or

REM cmd
dotnet vstest tests.dll -- TestRunParameters.Parameter(name=\"Browser\",value=\"Chrome\")

We will be working on improving this, but can’t promise any date.

@nohwnd Triple backslash confirmed working.

But I really do hope this feature get some more love? I don’t have anything good to say about the workaround other than it works.

Oh I see it. It also passes it to msbuild which then passes it to vstest.console via a msbuild parameter for the vstest task. The problem there is that the pass mechanism already puts the string into double quotes so we can’t simple double the single quotes to give msbuild a literal string (which is probably what we should do to avoid some of these quotest shenenigans). So we need to triply escape this.

dotnet test --settings local.runsettings -- 'TestRunParameters.Parameter(name=\\\"myparam\\\", value=\\\"foo\\\")'

Triple escape:

1. pass powershell parser by making the param a non-expandable string by putting single quotes around it, this will cause that the string will not be split on spaces, but it will get the quotes stripped.

PS> dotnet ConsoleApp18.dll 'a b'
args
a b
PS> dotnet ConsoleApp18.dll '"a b"'
args
a b

1. Escape quotes to get them through the command line parser while passed to dotnet.exe (or any other console).

PS> dotnet ConsoleApp18.dll ‘"a b"’ args “a b”

1. escape the double quotes that are passed to msbuild to ensure we don’t break the msbuild command by escaping every quote with \. That is why we need to end up with \" rather than just ".

"dotnet.exe" exec "...\MSBuild.dll" 
    -verbosity:normal "-property:VSTestCLIRunSettings=\"TestRunParameters.Parameter(name=\"myparam\", value=\"foo\")\"" 

PS> dotnet ConsoleApp18.dll '\\\"a b\\\"'
args
\"a b\"

1. There is one more pass, but here the task probably escapes correctly before passing the call, because vstest console is correctly passed escaped quotes \", so we don’t need to do anthing.

@kbirger the xpath must resolve to an existing node. Otherwise it will try and create the node. Refer code here

A little ugly but you could do something along the lines of: dotnet test -- TestRunParameters/Parameter[contains(@name,'SETTINGNAME')]/@value=SETTINGVALUE

Left-hand-side of the ‘=’ can be xpath that needs to resolve to an existing attribute or element.

Note: that the xpath cannot contain ‘=’.

This gives an error:

Unhandled Exception: System.Xml.XmlException: The ‘/’ character, hexadecimal value 0x2F, cannot be included in a name. at System.Xml.XmlDocument.CheckName(String name) at System.Xml.XmlElement…ctor(XmlName name, Boolean empty, XmlDocument doc) at System.Xml.XmlDocument.CreateElement(String prefix, String localName, String namespaceURI) at System.Xml.XmlDocument.CreateElement(String name) at Microsoft.VisualStudio.TestPlatform.Common.Utilities.RunSettingsProviderExtensions.CreateNode(XmlDocument doc, String xPath) at Microsoft.VisualStudio.TestPlatform.Common.Utilities.RunSettingsProviderExtensions.UpdateRunSettingsNode(IRunSettingsProvider runSettingsProvider, String key, String data) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.CLIRunSettingsArgumentExecutor.CreateOrOverwriteRunSettings(IRunSettingsProvider runSettingsProvider, String[] args) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.CLIRunSettingsArgumentExecutor.Initialize(String[] arguments) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass21_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0() at System.Lazy1.PublicationOnlyViaFactory(LazyHelper initializer) at System.Lazy1.CreateValue() at System.Lazy1.get_Value() at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.GetArgumentProcessors(String[] args, List1& processors) at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args) at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)