grpc: Grpc.Tools: Accented characters in nuget path breaks c# grpc tool compiler

@joergbattermann #22916 should fix this for you once merged as german umlauts are fine with UTF-8. I will try and ping Jan on that. I might need to rebase the PR.

@jtattermusch I think I have a clearer picture of the problem now. https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/compiler/command_line_interface.cc#L1326 reads the file using std::ifstream only which does not support multi-characters and BOM in files. This is also why it works on command line but not with command file.

We should use std::wifstream and make sure we also support BOM to get this fixed.

https://github.com/protocolbuffers/protobuf/pull/10200 was merged, which should fix this problem once

• the fix is released with the next version of protobuf
• grpc’s third_party/protobuf submodule is upgraded to the new protobuf version.

@jtattermusch The problem lies partly within protoc (non-UTF8 characters) and partly in how windows handles response file commands (which MSBuild uses). For my test, I renamed the grpc_csharp_plugin.exe to ágrpc_csharp_plugin.exe. I can execute the following command nicely on CMD directly but not through an RSP file:

test.rsp:

--plugin=protoc-gen-grpc=<path_to_protoc_in_nuget>\ágrpc_csharp_plugin.exe
--grpc_out=grpc_out
test.proto

Cmd: "<userprofile>\.nuget\packages\grpc.tools\2.28.1\tools\windows_x64\protoc.exe" @test.rsp

As soon as a non-ASCII character is in the RSP file, it does fail. And UTF-16 characters also seem to fail in protoc on CMD additionally… So chinese usernames fail for this reason either way.

EDIT: I found a way to change how we encode the response file here: https://github.com/grpc/grpc/blob/master/src/csharp/Grpc.Tools/ProtoCompile.cs#L375-L376

According to https://github.com/microsoft/msbuild/issues/4904 this should work on MSBuild 16.5.

Sadly, protoc only accepts UTF-8 without BOM. All other encodings (including Unicode and UTF-32 LE/BE & BOM/no-BOM) fail. I don’t understand why it’s not accepting á as this should be covered by UTF-8. But I was also not able to get it working with UTF-8 on CMD manually.

It seems appveyor may not be a good choice for this kind of tests, they also have problem processing non-ascii characters. You may want to run a VM locally to test things.

Bump.

can you provide a simple repro of the protoc accented characters handling problem, so that we can followup with the protobuft team? Alternatively, feel free to file a protobuf issue directly.

@jtattermusch No, I think the problem is where the compiler is located, not the proto file itself. I can’t compile even the Greeter.proto before moving my global-packages path to D:\nuget_packages\packages\

I copied the example code from this repository at 9993d2b9a421a0b27b604c52fa93260fe0638b12 and made a POC using appveyor: GitHub repo and appveyor build.

Note: Currently the build with NUGET_PACKAGES=C:\啊あ\packages is failing because Illegal characters in path when running nuget locals all -list, probably a appveyor bug because I have no problem running with the same setup on my local machine.

PS C:\> $env:NUGET_PACKAGES="C:\啦あ\packages"
PS C:\> nuget locals all -list
http-cache: C:\Users\[removed]\AppData\Local\NuGet\v3-cache
global-packages: C:\啦あ\packages
temp: C:\Users\[removed]\AppData\Local\Temp\NuGetScratch