puppeteer-sharp: Unrecoverable exceptions being frequently thrown

Description

Unrecoverable exceptions are being thrown by Puppeteer Sharp. This prevents any kind of retry or graceful exception handling and results in the process being terminated.

Complete minimal example reproducing the issue

    // Arrange
            var launchOptions = new LaunchOptions()
            {
                Headless = true,
            };

            var viewPortOptions = new ViewPortOptions()
            {
                Width = 1920,
                Height = 1080
            };

            var sites = new List<string>()
            {
                "wix.com",
                "facebook.com",
                "twitter.com",
                "google.com",
                "youtube.com",
                "instagram.com",
                "linkedin.com",
                "wordpress.org",
                "pinterest.com",
                "wikipedia.org",
                "wordpress.com",
                "blogspot.com",
                "apple.com",
                "adobe.com",
                "tumblr.com",
            }
    // Act
            try
            {
                await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
                using (var browser = await Puppeteer.LaunchAsync(launchOptions))
                {
                    //var context = await browser.CreateIncognitoBrowserContextAsync();
                    //var page = await context.NewPageAsync();
                    var page = await browser.NewPageAsync();
                    await page.SetViewportAsync(viewPortOptions);

                    foreach (var site in sites)
                    {
                        try
                        {
                            await page.GoToAsync($"http://{site}");
                            Console.WriteLine(await page.GetTitleAsync());
                            await page.ScreenshotAsync($"D:\\bin\\screenshots\\{site}.png");
                        }
                        catch (Exception exception)
                        {
                            // Catches most exceptions such as timeouts but does not catch others, see below.
                            Console.WriteLine($"Unable to take screenshot of: {site}. Exception: {exception.Message}");
                        }
                    }
                }
            }
            catch (Exception exception)
            {
                // Never enters the catch.
                Console.WriteLine($"Unable to proceed: {exception.Message}");
            }

Expected behavior:

The library to take a screenshot of the URL given.

Actual behavior:

Often crashes with exceptions which cannot be caught. There appears to be no common cause of the issue and it happens at random. The issue does not seem to be related to a particular site.

Same happens with normal and Incognito sessions.

Versions

Which version of PuppeteerSharp are you using? 1.8.0
Which .NET runtime and version are you targeting? .NET Core 2.1

Additional Information

The exceptions are lacking in details and so far the exceptions encountered are:

PuppeteerSharp.TargetCrashedException
  HResult=0x80131500
  Source=PuppeteerSharp
  StackTrace:
   at PuppeteerSharp.Page.OnTargetCrashed()
   at PuppeteerSharp.Page.<Client_MessageReceived>d__199.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

System.Collections.Generic.KeyNotFoundException
  HResult=0x80131577
  Message=The given key 'B99316384B7CE098D1E12423B7A2772B' was not present in the dictionary.
  Source=System.Private.CoreLib
  StackTrace:
   at System.ThrowHelper.ThrowKeyNotFoundException[T](T key)
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at PuppeteerSharp.NetworkManager.OnRequest(RequestWillBeSentPayload e, String interceptionId)
   at PuppeteerSharp.NetworkManager.OnRequestWillBeSent(RequestWillBeSentPayload e)
   at PuppeteerSharp.NetworkManager.<Client_MessageReceived>d__33.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Here is the redirected output from Chrome. Again because this happens randomly I haven’t been able to determine an exact reproduction but it seems to happen more often when it navigates to a site that does not exist.

Unable to take screenshot of: wixsite.com. Exception: net::ERR_NAME_NOT_RESOLVED
 at http://wixsite.com at http://wixsite.com
Received fatal exception EXCEPTION_ACCESS_VIOLATION
Backtrace:
        GetHandleVerifier [0x00007FF9B8058D94+15468820]
        ovly_debug_event [0x00007FF9B6A08B5D+16084829]
        ovly_debug_event [0x00007FF9B6A088B7+16084151]
        ovly_debug_event [0x00007FF9B69FC185+16033157]
        ovly_debug_event [0x00007FF9B69FBA36+16031286]
        ovly_debug_event [0x00007FF9B69F764B+16013899]
        GetHandleVerifier [0x00007FF9B80583D0+15466320]
        GetHandleVerifier [0x00007FF9B77C89CC+6489420]
        GetHandleVerifier [0x00007FF9B77C8560+6488288]
        GetHandleVerifier [0x00007FF9B77C9A01+6493569]
        ovly_debug_event [0x00007FF9B626E19A+8111514]
        GetHandleVerifier [0x00007FF9B72C4E0F+1231247]
        GetHandleVerifier [0x00007FF9B72C3384+1224452]
        IsSandboxedProcess [0x00007FF9B71781BC+1557308]
        IsSandboxedProcess [0x00007FF9B718C3E6+1639782]
        IsSandboxedProcess [0x00007FF9B71781BC+1557308]
        ovly_debug_event [0x00007FF9B6CC0C4F+18935887]
        ovly_debug_event [0x00007FF9B6CC1045+18936901]
        IsSandboxedProcess [0x00007FF9B717AAF9+1567865]
        ovly_debug_event [0x00007FF9B6CD96D1+19036881]
        IsSandboxedProcess [0x00007FF9B713319E+1274654]
        ovly_debug_event [0x00007FF9B6C9A9D3+18779603]
        ovly_debug_event [0x00007FF9B6C9E9C6+18795974]
        ovly_debug_event [0x00007FF9B6C9A191+18777489]
        IsSandboxedProcess [0x00007FF9B716F27D+1520637]
        IsSandboxedProcess [0x00007FF9B716F194+1520404]
        ovly_debug_event [0x00007FF9B6C9CCAE+18788526]
        ChromeMain [0x00007FF9B5AB1111+273]
        Ordinal0 [0x00007FF7F23531CC+12748]
        Ordinal0 [0x00007FF7F235168D+5773]
        GetHandleVerifier [0x00007FF7F24557E2+793282]
        BaseThreadInitThunk [0x00007FFA016A13D2+34]
        RtlUserThreadStart [0x00007FFA036354F4+52]

As a side effect dozens of Chromium instances are left running, though that’s not really within scope of this issue.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 32 (16 by maintainers)

Most upvoted comments

I’ve run a few hours of testing and the issue seems to have been resolved and no process termination 👍 Thanks!

I have however come across a new edge case. I’m not sure if it belongs in this issue or a new one though.

When in headless mode navigating to http://one.com causes the Windows/Console beep to occur - every time the site is hit (I’ve not encountered this on any other site). When not running in headless mode and hitting the same site and with SlowMo enabled, however, the console beeps are not emitted and I get the following exceptions (not in any particular order).

Let me know if this is better suited in a new issue!

System.InvalidOperationException
  HResult=0x80131509
  Message=Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
  Source=System.Private.CoreLib
  StackTrace:
   at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported()
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at PuppeteerSharp.Browser.<CreateTargetAsync>d__63.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Browser.cs:line 399
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at PuppeteerSharp.Browser.<Connect_MessageReceived>d__60.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Browser.cs:line 335

System.InvalidOperationException
  HResult=0x80131509
  Message=An attempt was made to transition a task to a final state when it had already completed.
  Source=System.Private.CoreLib
  StackTrace:
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at PuppeteerSharp.Frame.SetDefaultContext(ExecutionContext context) in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Frame.cs:line 644
   at PuppeteerSharp.FrameManager.<OnExecutionContextCreatedAsync>d__44.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 247
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at PuppeteerSharp.FrameManager.<_client_MessageReceived>d__39.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 178

System.IndexOutOfRangeException
  HResult=0x80131508
  Message=Index was outside the bounds of the array.
  Source=System.Collections
  StackTrace:
   at System.Collections.Generic.HashSet`1.Enumerator.MoveNext()
   at PuppeteerSharp.FrameManager.AddFrame(String framId, Frame frame) in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 305
   at PuppeteerSharp.FrameManager.OnFrameNavigated(FramePayload framePayload) in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 292
   at PuppeteerSharp.FrameManager.<_client_MessageReceived>d__39.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 162

System.IndexOutOfRangeException
  HResult=0x80131508
  Message=Index was outside the bounds of the array.
  Source=System.Collections
  StackTrace:
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
   at System.Collections.Generic.HashSet`1.Add(T item)
   at PuppeteerSharp.Helpers.MultiMap`2.Add(TKey key, TValue value) in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Helpers\MultiMap.cs:line 15
   at PuppeteerSharp.FrameManager.<GetFrameAsync>d__54.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 376
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at PuppeteerSharp.FrameManager.<OnExecutionContextCreatedAsync>d__44.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 236
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at PuppeteerSharp.FrameManager.<_client_MessageReceived>d__39.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 178

System.IndexOutOfRangeException
  HResult=0x80131508
  Message=Index was outside the bounds of the array.
  Source=System.Collections
  StackTrace:
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
   at System.Collections.Generic.HashSet`1.Add(T item)
   at PuppeteerSharp.Helpers.MultiMap`2.Add(TKey key, TValue value) in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Helpers\MultiMap.cs:line 15
   at PuppeteerSharp.FrameManager.<GetFrameAsync>d__54.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 376
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at PuppeteerSharp.FrameManager.<OnExecutionContextCreatedAsync>d__44.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 236
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at PuppeteerSharp.FrameManager.<_client_MessageReceived>d__39.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\FrameManager.cs:line 178

System.NullReferenceException
  HResult=0x80004003
  Message=Object reference not set to an instance of an object.
  Source=System.Private.CoreLib
  StackTrace:
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.ContainsKey(TKey key)
   at PuppeteerSharp.Browser.<CreateTargetAsync>d__63.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Browser.cs:line 394
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at PuppeteerSharp.Browser.<Connect_MessageReceived>d__60.MoveNext() in C:\Users\Lloyd\Source\Repos\lloydjatkinson\puppeteer-sharp\lib\PuppeteerSharp\Browser.cs:line 335

lloydjatkinson on Nov 1, 2018

Could you pull from support/onrequest-changes again @lloydjatkinson?

kblok on Oct 31, 2018

According to https://github.com/dotnet/roslyn/issues/13897#issuecomment-248098377

async void with an unhandled exception brings down the process

These are the two issues, I believe, we have. We are doing async void to process WebSocket messages. If some method fails there, the entire process goes down. The second problem is that if we call async void events using Event?.Invoke those async methods become fire-and-forget tasks. So the problem we might be getting is that, when we get a targetCreated event, we invoke it, but then we start invoking other events (that need the new target) while the task creating the target didn’t finish yet.

cc @Meir017

kblok on Oct 30, 2018

@lloydjatkinson I think I found the root of these issues, but it requires some refactors. I’m working on it.

kblok on Oct 30, 2018

Could you test this branch?
https://github.com/kblok/puppeteer-sharp/pull/720

kblok on Oct 28, 2018