Issue Title

The Unload method does not free any assemblies returned from LoadFromStream that are still referenced somewhere.

General

netcoreapp3.1

        public class TestAssemblyLoader : AssemblyLoadContext, IAssemblyLoader
        {
            public TestAssemblyLoader() : base(true)
            {
                LoadedAssemblies = new Dictionary<string, Assembly>();

                Resolving += LoadContext_Resolving;
            }

            private Assembly LoadContext_Resolving(AssemblyLoadContext arg1, AssemblyName arg2)
            {
                return LoadedAssemblies[arg2.FullName];
            }

            private Dictionary<string, Assembly> LoadedAssemblies { get; }

            Assembly IAssemblyLoader.LoadFromStream(Stream stream)
            {
                var assembly = LoadFromStream(stream);
                return LoadedAssemblies[assembly.FullName] = assembly;
            }

            protected sealed override Assembly Load(AssemblyName assemblyName)
            {
                return null;
            }

            public void Dispose()
            {
                LoadedAssemblies.Clear(); // Without this call, the assemblies in this dictionary don't get unloaded.
                Unload();
            }
        }

Symptom is visible in unreleased file handles - example repro here: https://github.com/dave-yotta/roslyn-assemblyunload

It might make sense for this to be expected behaviour, but I certainly didn’t expect it. (nor did our production machines 😢)

At least can there be an argument to unload like bool throwIfAnyUnloadFailed = false so that it’s self-documenting to callers somehow?

Related issue I’m coming from here: https://github.com/dotnet/roslyn/issues/49282