roslyn: Sporadic failures in AbstractPersistentStorageTests

Hitting some failures in AbstractPersistentStorageTests. It’s no ta single failure but instead roughly the same stack trace spread out across a number of tests. The rough stack is the following

[xUnit.net 00:00:27.63]     Microsoft.CodeAnalysis.UnitTests.WorkspaceServices.CloudCachePersistentStorageTests.TestReadChecksumProducesLastWrittenChecksum1 [FAIL]
  Failed Microsoft.CodeAnalysis.UnitTests.WorkspaceServices.CloudCachePersistentStorageTests.TestReadChecksumProducesLastWrittenChecksum1 [1 s]
  Error Message:
   Assert.False() Failure
Expected: False
Actual:   True
  Stack Trace:
     at Microsoft.CodeAnalysis.UnitTests.WorkspaceServices.AbstractPersistentStorageTests.<TestReadChecksumProducesLastWrittenChecksum1>d__42.MoveNext() in /_/src/VisualStudio/CSharp/Test/PersistentStorage/AbstractPersistentStorageTests.cs:line 392
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.CodeAnalysis.UnitTests.WorkspaceServices.AbstractPersistentStorageTests.<TestReadChecksumProducesLastWrittenChecksum1>d__42.MoveNext() in /_/src/VisualStudio/CSharp/Test/PersistentStorage/AbstractPersistentStorageTests.cs:line 392
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:39.47]     Microsoft.VisualStudio.LanguageServices.CSharp.UnitTests.CodeModel.FileCodeFunctionTests.Overloads_Count [SKIP]
  Skipped Microsoft.VisualStudio.LanguageServices.CSharp.UnitTests.CodeModel.FileCodeFunctionTests.Overloads_Count [1 ms]
[xUnit.net 00:01:41.67]     Microsoft.CodeAnalysis.Editor.CSharp.UnitTests.Interactive.Commands.InteractiveCommandHandlerTests.TestExecuteInInteractiveWithDefines [SKIP]
  Skipped Microsoft.CodeAnalysis.Editor.CSharp.UnitTests.Interactive.Commands.InteractiveCommandHandlerTests.TestExecuteInInteractiveWithDefines [1 ms]
Results File: C:\h\w\AD0F09A7\w\A60208F7\e\Microsoft.VisualStudio.LanguageServices.CSharp.UnitTests.dll_net472_x86_test_results.xml

Example build:

Build	Kind	Start Time
1128675	Rolling	2021-08-05
1133858	Rolling	2021-12-05

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 24 (24 by maintainers)

Most upvoted comments

Ok, got a repro! It depends on the cache being flushed to disk between the two write operations.

AArnott on May 12, 2021

I have the fix ready to go into 16.10, and a PR to send it to 17.0 right now that is just waiting on a code review.

I think it’s fine for this to go into 17.0. Thanks!

CyrusNajmabadi on May 17, 2021

our wrapper over you is very thin. you can see it here: https://github.com/dotnet/roslyn/blob/main-vs-deps/src/VisualStudio/Core/Def/Storage/CloudCachePersistentStorage.cs#L179

lmk if anything looks innappropriate.

CyrusNajmabadi on May 12, 2021

ICacheService.SetItemAsync only completes its returned task after writing has been committed. ICacheService.CheckExistsAsync has no cache of its own, and always has to obtain that read scheduler’s time to read the db before returning the result.

However I see here that we only initialize the version (aka checksum) column when the checksum is non-null. So I suspect the issue here may be a stale value is retained by the sqlite3_stmt object that we reuse, so we effectively are re-applying a stale value. I will investigate.

AArnott on May 12, 2021