webdis: Inconsistent behavior between GET / POST

Great! I’m glad it worked for you too, I was worried the internal field names would be different on .NET for macOS for whatever reason – as long as they’re not public we can’t really assume much.

It was interesting to have a quick look at modern .NET and see how much it has changed since I last used it seriously in 2006-2007. I had actually tried to do this around the time of my earlier message identifying the issue as coming from the HTTP client, and went down the wrong path flipping the bit for a different flag which made no difference for the /./ use case. Good thing I gave it a second try 😃

By the way, I learned that even my tool of choice for crafting HTTP requests – the ubiquitous curl – simplifies request URIs by default unless it’s run with --path-as-is.

I think we can close this issue now, the subject has drifted quite a bit but as far as I can tell all the weird behaviors with Postman and the .NET HTTP client have been explained and workarounds were found for both.

Let me know what you think!

Above and beyond bro! I was hitting dead ends because it was so embedded into the .NET, and as you pointed out .NET 7 is a different beast. I have tested this and it works well. Thank you!

Okay, I think I have a solution for you. Please read carefully.

There are many threads online about how to prevent the .NET URI parser from simplifying URIs in this manner. This is sometimes referred to as “normalizing”, sometimes “canonicalizing”. It is done by the UriParser class in .NET, based on behavior flags provided to it in its constructor.

If you go to System.UriSyntaxFlags, you’ll find a list of these flags. They are “OR’d” together as binary flags, and separate instances of UriParser are available by default for various protocols. If you go to the code for System.UriParser, you’ll find them:

    public abstract partial class UriParser
    {
        internal static readonly UriParser HttpUri = new BuiltInUriParser("http", 80, HttpSyntaxFlags);
        internal static readonly UriParser HttpsUri = new BuiltInUriParser("https", 443, HttpUri._flags);
        internal static readonly UriParser WsUri = new BuiltInUriParser("ws", 80, HttpSyntaxFlags);
...

There are many more. Note how HttpSyntaxFlags is passed to the "http" parser. It is defined as:

        private const UriSyntaxFlags HttpSyntaxFlags =
                                        // [...] truncated for brevity, there are many more values
                                        UriSyntaxFlags.ConvertPathSlashes |
                                        UriSyntaxFlags.CompressPath |
                                        UriSyntaxFlags.CanonicalizeAsFilePath |
                                        UriSyntaxFlags.AllowIdn |
                                        UriSyntaxFlags.AllowIriParsing;

These flags – and especially CompressPath – control the behavior of the UriParser instance for HTTP URIs. They are themselves part of an enum, once again we can look it up and find the values (truncated here for brevity):

  internal enum UriSyntaxFlags
  {
    None = 0,
    MustHaveAuthority = 1,
    OptionalAuthority = 2,
// ... many more ...
    CompressPath = 8388608, // 0x00800000
    CanonicalizeAsFilePath = 16777216, // 0x01000000
    UnEscapeDotsAndSlashes = 33554432, // 0x02000000
// ... again some more ...
  }

All of this is private and not externally configurable, except… if we use reflection. I wrote that there are lots of threads about it online, and they all bring up some version of this technique. For example:

• StackOverflow: Uri canonicalization compacting FTP scheme
• StackOverflow: System.Net.Uri with urlencoded characters
• StackOverflow: How to make System.Uri not to unescape %2f (slash) in path?
• Apparently an official workaround from Microsoft themselves
• Lots of code on GitHub with the exact same trick

The problem with all of these is that they are only valid for old versions of .NET, and they no longer work. None of them do. Look at the dates: 2010, 2011, 2012… this is code that modifies the internal fields of UriParser by name, so if they were renamed at some point or if the flags were changed, it’s not going to work. And that’s exactly what happened over time.

What we need to do is this:

1. For both "http" and "https", get the dedicated UriParser object
2. Get a reference to the _flags field of the UriParser class (it’s not longer m_Flags like in 2010)
3. Extract the value for that field from the UriParser object we got above
4. Convert it to an int
5. Unset whatever flags we don’t want by AND’ing them with NOT(the value) (see here for details)
6. Saving the new value to the _flags field of our current UriParser object

I tried doing all this and it seems to work well. I do see requests being sent to Webdis with all their /./././ still present when I first make these changes.

Here’s a screenshot of my Rider debugger showing the structure or uriParser (the UriParser instance for `“http” in this case):

• (1) is the uriParser object
• (2) is its Flags getter
• (3) is its _flags field that’s returned by the Flags getter

There are more values than fit on the screen, this is the full list:

MustHaveAuthority | MayHaveUserInfo | MayHavePort | MayHavePath | MayHaveQuery | MayHaveFragment
 | AllowUncHost | AllowDnsHost | AllowIPv4Host | AllowIPv6Host | AllowAnInternetHost | SimpleUserSyntax
 | BuiltInSyntax | PathIsRooted | ConvertPathSlashes | CompressPath | CanonicalizeAsFilePath | AllowIdn
 | AllowIriParsing

Here is the method to make these changes, call it at the start of your program:

    private static void DoNotSimplifyUris()
    {
        const int CanonicalizeAsFilePath = 0x01000000;
        const int ConvertPathSlashes = 0x00400000;
        const int CompressPath = 0x00800000; // only this one seems strictly necessary for the /././…  conversion.
        
        var getSyntaxMethod = typeof (UriParser).GetMethod("GetSyntax", BindingFlags.Static | BindingFlags.NonPublic);
        if (getSyntaxMethod == null)
        {
            throw new MissingMethodException("UriParser", "GetSyntax");
        }

        foreach (var scheme in new[] { "http", "https" })
        {
            // call with "http" and "https" to update both UriParser objects (see UriParser class for all instances)
            var uriParser = getSyntaxMethod.Invoke(null, new object[] { scheme });
            if (uriParser == null)
            {
                throw new ArgumentNullException($"Unexpected: UriParser.getSyntax({scheme}) returned null");
            }

            // get reference to UriParser._flags field
            var flagsFieldInfo = typeof(UriParser).GetField("_flags",
                BindingFlags.NonPublic | BindingFlags.GetField | BindingFlags.SetField | BindingFlags.Instance);
            if (flagsFieldInfo == null)
            {
                throw new MissingFieldException("UriParser", "_flags");
            }

            // get value of that field on the UriParser object we're looking at (either the http or https instance)
            var flagsValue = flagsFieldInfo.GetValue(uriParser);
            if (flagsValue == null)
            {
                throw new Exception($"Could not extract the value of UriParser._flags for the {scheme} instance");
            }

            // convert to the underlying int representation to unset some flags
            var flags = (int) flagsValue;
            flags &= ~CanonicalizeAsFilePath;
            flags &= ~ConvertPathSlashes;
            flags &= ~CompressPath;

            // save the modified value on the UriParser instance
            flagsFieldInfo.SetValue(uriParser, flags);
        }
    }

I’ve also written a test program that does this:

1. Calls DoNotSimplifyUris()
2. Sends a request to Webdis on /SET/foo/bar
3. Sends a request to Webdis on /GET/foo/bar
4. Sends a request to Webdis on /GET/././././foo – I see it unchanged in the Webdis logs
5. Sends a request to Webdis on /GET/././././foo, but this time using a Uri object instead of a string – again, I see it unchanged in the Webdis logs

Here is the full program: https://gist.github.com/nicolasff/5de66eda4264a27637aebe2e767e4abf

This is its output:

'SET foo bar' returned: {"SET":[true,"OK"]}
'GET foo' returned: {"GET":"bar"}
'GET . . . . foo' returned: {"GET":[false,"ERR wrong number of arguments for 'get' command"]}
(manual) 'GET . . . . foo' (using Uri instead of String) returned: HTTP 200 OK
(manual) response body: {"GET":[false,"ERR wrong number of arguments for 'get' command"]}

and if I comment the initial call to DoNotSimplifyUris():

'SET foo bar' returned: {"SET":[true,"OK"]}
'GET foo' returned: {"GET":"bar"}
'GET . . . . foo' returned: {"GET":"bar"}
(manual) 'GET . . . . foo' (using Uri instead of String) returned: HTTP 200 OK
(manual) response body: {"GET":"bar"}

^ clearly the /GET/././././foo get simplified to /GET/foo.

It would be nice if Microsoft made this configurable, but as you can see it’s been at least 12-13 years and they themselves have suggested this reflection hack on .NET support forums so it doesn’t look like a cleaner solution is coming any time soon. Note that this is a .NET 7 version, I don’t know if it would work with other versions and there’s no guarantee it’ll continue to work. I thought it was worth posting here since I couldn’t find a version of it for recent releases of .NET.

Let me know what you think!