runtime: Breaking change in netcoreapp2.1 (possibly JIT bug?) re fixed buffers

New in netcoreapp2.1 (doesn’t impact netcoreapp2.0 or net47)

Full repro below; synopsis: fixed buffers go pretty wild when used on a field in a key in a Dictionary<TKey,TValue> (and possibly some other scenarios), causing:

• items not being found even though it can be shown (by looking via foreach) that GetHashCode() and Equals() are working correctly
• data corruption - in particular when more than one element in the fixed buffer is involved (so… most of the time; a fixed buffer of length 1 isn’t very useful)

Since this could affect many more scenarios, I’d wager this is a pretty serious regression.

Code:

// #define EXPLICIT_FIELDS

// Hypothesis: JIT bug affecting how fixed buffers are handled
//
// Affects: netcoreapp2.1, debug and release
// Does not seem to affect: netcoreapp2.0, net47
//
// the idea behind CommandBytes is that it is a fixed-sized string-like thing
// used for matching commands; it is *implemented* as a fixed buffer
// of **longs**, but: the first byte of the first element is coerced into
// a byte and used to store the length; the actual text payload (ASCII)
// starts at the second byte of the first element
//
// as far as I can tell, it is all validly implemented, and it works fine
// in isolation, however: when used in a dictionary, it goes bad;
// - items not being found despite having GetHashCode and Equals match
// - items over 1 chunk size becoming corrupted (see: ToInnerString)
//
// however, if I replace the fixed buffer with the same number of
// regular fields (_c0,_c1,_c2) and use *all the same code*, it
// all works correctly! to see this, define EXPLICIT_FIELDS
//
// The "Main" method populates a dictionary in the expected way,
// then attempts to find things - either via TryGetValue or manually;
// it then compares the contents
//
// Yes, this code is evil; it is for a very specific optimized scenario.

using System;
using System.Collections.Generic;
using System.Text;

unsafe struct CommandBytes : IEquatable<CommandBytes>
{
    private const int ChunkLength = 3;
    public const int MaxLength = (ChunkLength * 8) - 1;

#if !EXPLICIT_FIELDS
    fixed long _chunks[ChunkLength];
#else
    long _c0, _c1, _c2;
#endif
    public override int GetHashCode()
    {
#if !EXPLICIT_FIELDS
        fixed (long* lPtr = _chunks)
#else
        fixed (long* lPtr = &_c0)
#endif
        {
            var hashCode = -1923861349;
            long* x = lPtr;
            for (int i = 0; i < ChunkLength; i++)
            {
                hashCode = hashCode * -1521134295 + (*x++).GetHashCode();
            }
            return hashCode;
        }
    }

    public override string ToString()
    {
#if !EXPLICIT_FIELDS
        fixed (long* lPtr = _chunks)
#else
        fixed (long* lPtr = &_c0)
#endif
        {
            var bPtr = (byte*)lPtr;
            return Encoding.ASCII.GetString(bPtr + 1, bPtr[0]);
        }
    }
    public int Length
    {
        get
        {
#if !EXPLICIT_FIELDS
            fixed (long* lPtr = _chunks)
#else
            fixed (long* lPtr = &_c0)
#endif
            {
                var bPtr = (byte*)lPtr;
                return bPtr[0];
            }
        }
    }
    public byte this[int index]
    {
        get
        {
#if !EXPLICIT_FIELDS
            fixed (long* lPtr = _chunks)
#else
            fixed (long* lPtr = &_c0)
#endif
            {
                byte* bPtr = (byte*)lPtr;
                int len = bPtr[0];
                if (index < 0 || index >= len) throw new IndexOutOfRangeException();
                return bPtr[index + 1];
            }
        }
    }

    public CommandBytes(string value)
    {
        value = value.ToLowerInvariant();
        var len = Encoding.ASCII.GetByteCount(value);
        if (len > MaxLength) throw new ArgumentOutOfRangeException("Maximum command length exceeed");

#if !EXPLICIT_FIELDS
        fixed (long* lPtr = _chunks)
#else
        _c0 = _c1 = _c2 = 0L;
        fixed (long* lPtr = &_c0)
#endif
        {
            Clear(lPtr);
            byte* bPtr = (byte*)lPtr;
            bPtr[0] = (byte)len;
            fixed (char* cPtr = value)
            {
                Encoding.ASCII.GetBytes(cPtr, value.Length, bPtr + 1, len);
            }
        }
    }
    public override bool Equals(object obj) => obj is CommandBytes cb && Equals(cb);

    public string ToInnerString()
    {
#if !EXPLICIT_FIELDS
        fixed (long* lPtr = _chunks)
#else
        fixed (long* lPtr = &_c0)
#endif
        {
            long* x = lPtr;
            var sb = new StringBuilder();
            for (int i = 0; i < ChunkLength; i++)
            {
                if (sb.Length != 0) sb.Append(',');
                sb.Append(*x++);
            }
            return sb.ToString();
        }
    }
    public bool Equals(CommandBytes value)
    {
#if !EXPLICIT_FIELDS
        fixed (long* lPtr = _chunks)
#else
        fixed (long* lPtr = &_c0)
#endif
        {
            long* x = lPtr;
#if !EXPLICIT_FIELDS
            long* y = value._chunks;
#else
            long* y = &value._c0;
#endif
            for (int i = 0; i < ChunkLength; i++)
            {
                if (*x++ != *y++) return false;
            }
            return true;
        }
    }
    private static void Clear(long* ptr)
    {
        for (int i = 0; i < ChunkLength; i++)
        {
            *ptr++ = 0L;
        }
    }
}

static class Program
{
    static void Main()
    {
        var lookup = new Dictionary<CommandBytes, string>();
        void Add(string val)
        {
            var cb = new CommandBytes(val);
            // prove we didn't screw up
            if (cb.ToString() != val)
                throw new InvalidOperationException("oops!");
            lookup.Add(cb, val);
        }
        Add("client");
        Add("cluster");
        Add("command");
        Add("config");
        Add("dbsize");
        Add("decr");
        Add("del");
        Add("echo");
        Add("exists");
        Add("flushall");
        Add("flushdb");
        Add("get");
        Add("incr");
        Add("incrby");
        Add("info");
        Add("keys");
        Add("llen");
        Add("lpop");
        Add("lpush");
        Add("lrange");
        Add("memory");
        Add("mget");
        Add("mset");
        Add("ping");
        Add("quit");
        Add("role");
        Add("rpop");
        Add("rpush");
        Add("sadd");
        Add("scard");
        Add("select");
        Add("set");
        Add("shutdown");
        Add("sismember");
        Add("spop");
        Add("srem");
        Add("strlen");
        Add("subscribe");
        Add("time");
        Add("unlink");
        Add("unsubscribe");

        void HuntFor(string lookFor)
        {
            Console.WriteLine($"Looking for: '{lookFor}'");
            var hunt = new CommandBytes(lookFor);


            if (lookup.TryGetValue(hunt, out var found))
            {
                Console.WriteLine($"Found via TryGetValue: '{found}'");
            }
            else
            {
                Console.WriteLine("**NOT FOUND** via TryGetValue");
            }
            Console.WriteLine("looking manually");
            foreach (var pair in lookup)
            {
                if (pair.Value == lookFor)
                {
                    Console.WriteLine($"Found manually: '{pair.Value}'");
                    var key = pair.Key;
                    void Compare<T>(string caption, Func<CommandBytes, T> func)
                    {
                        T x = func(hunt), y = func(key);
                        Console.WriteLine($"{caption}: {EqualityComparer<T>.Default.Equals(x, y)}, '{x}' vs '{y}'");
                    }
                    Compare("GetHashCode", _ => _.GetHashCode());
                    Compare("ToString", _ => _.ToString());
                    Compare("Length", _ => _.Length);
                    Compare("ToInnerString", _ => _.ToInnerString());
                    Console.WriteLine($"Equals: {key.Equals(hunt)}, {hunt.Equals(key)}");
                    var eq = EqualityComparer<CommandBytes>.Default;

                    Console.WriteLine($"EqualityComparer: {eq.Equals(key, hunt)}, {eq.Equals(hunt, key)}");
                    Compare("eq GetHashCode", _ => eq.GetHashCode(_));
                }
            }
            Console.WriteLine();
        }
        HuntFor("ping");
        HuntFor("subscribe");
    }
}

Expected output:

Looking for: 'ping'
Found via TryGetValue: 'ping'
looking manually
Found manually: 'ping'
GetHashCode: True, '-1991953770' vs '-1991953770'
ToString: True, 'ping' vs 'ping'
Length: True, '4' vs '4'
ToInnerString: True, '444234035204,0,0' vs '444234035204,0,0'
Equals: True, True
EqualityComparer: True, True
eq GetHashCode: True, '-1991953770' vs '-1991953770'

Looking for: 'subscribe'
Found via TryGetValue: 'subscribe'
looking manually
Found manually: 'subscribe'
GetHashCode: True, '764962383' vs '764962383'
ToString: True, 'subscribe' vs 'subscribe'
Length: True, '9' vs '9'
ToInnerString: True, '7598244868551701257,25954,0' vs '7598244868551701257,25954,0'
Equals: True, True
EqualityComparer: True, True
eq GetHashCode: True, '764962383' vs '764962383'

(note Found via TryGetValue, and lots of True indicating no corruption)

Actual output on netcoreapp2.1

Looking for: 'ping'
**NOT FOUND** via TryGetValue
looking manually
Found manually: 'ping'
GetHashCode: True, '-1991953770' vs '-1991953770'
ToString: True, 'ping' vs 'ping'
Length: True, '4' vs '4'
ToInnerString: True, '444234035204,0,0' vs '444234035204,0,0'
Equals: True, True
EqualityComparer: True, True
eq GetHashCode: True, '-1991953770' vs '-1991953770'

Looking for: 'subscribe'
**NOT FOUND** via TryGetValue
looking manually
Found manually: 'subscribe'
GetHashCode: False, '764962383' vs '945069981'
ToString: False, 'subscribe' vs 'subscri  '
Length: True, '9' vs '9'
ToInnerString: False, '7598244868551701257,25954,0' vs '7598244868551701257,0,0'
Equals: False, False
EqualityComparer: False, False
eq GetHashCode: False, '764962383' vs '945069981'

Note the **NOT FOUND** for both; the first item doesn’t have data corruption (but fails anyway) - the second item (longer, hits 2 elements) has become corrupted.