dotnet-docs-samples: Gemini 1.5 Pro via Vertex AI API gives StatusCode="Unavailable" System.IO.IOException

Hello,

I’m using Google.Cloud.AIPlatform.V1 v2.27.0 to use Gemini 1.5 Pro via Vertex AI but usually I get System.IO.IOException.

https://github.com/GoogleCloudPlatform/dotnet-docs-samples/blob/main/aiplatform/api/AIPlatform.Samples/GeminiQuickstart.cs

This code is from the above example:

using PredictionServiceClient.StreamGenerateContentStream response = predictionServiceClient.StreamGenerateContent(generateContentRequest);

StringBuilder fullText = new();

AsyncResponseStream<GenerateContentResponse> responseStream = response.GetResponseStream();
await foreach (GenerateContentResponse responseItem in responseStream)
{

This is the exception:

Status(StatusCode="Unavailable", Detail="Error reading next message. IOException: The request was aborted. IOException: The response ended prematurely while waiting for the next frame from the server.", DebugException="System.IO.IOException: The request was aborted.")

The project is running on Ubuntu with .NET 7.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

I believe this is related to https://github.com/grpc/grpc-dotnet/issues/2358

Two workarounds - these are alternatives to each other:

  • You can specify an app switch to disable dynamic window sizing:
    • AppContext.SetSwitch("System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing", true);
  • You can use Grpc.Core instead of Grpc.Net.Client:
    • Add a dependency to Grpc.Core version 2.46.6
    • Add a using directive for Google.Api.Gax.Grpc
    • In the PredictionServiceClientBuilder object initializer, add GrpcAdapter = GrpcCoreAdapter.Instance

@jskeet I reproduced the issue @fatihyildizhan reported.

If you change the model name in GeminiQuickstart to this:

        string model = "gemini-1.5-pro-preview-0409" //"gemini-1.0-pro-vision"

And run the sample, it will fail most of the time with this error:

[xUnit.net 00:00:08.11]     GeminiQuickstartTest.TestGenerateContentAsync [FAIL]
  Failed GeminiQuickstartTest.TestGenerateContentAsync [7 s]
  Error Message:
   Grpc.Core.RpcException : Status(StatusCode="Unavailable", Detail="Error reading next message. IOException: The request was aborted. IOException: The response ended prematurely while waiting for the next frame from the server.", DebugException="System.IO.IOException: The request was aborted.")
---- System.IO.IOException : The request was aborted.
-------- System.IO.IOException : The response ended prematurely while waiting for the next frame from the server.
  Stack Trace:
     at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
   at Google.Api.Gax.Grpc.AsyncResponseStream`1.MoveNextAsync(CancellationToken cancellationToken)
   at GeminiQuickstart.GenerateContent(String projectId, String location, String publisher, String model) in /Users/atamel/dev/github/meteatamel/dotnet-docs-samples/aiplatform/api/AIPlatform.Samples/GeminiQuickstart.cs:line 82
   at GeminiQuickstart.GenerateContent(String projectId, String location, String publisher, String model) in /Users/atamel/dev/github/meteatamel/dotnet-docs-samples/aiplatform/api/AIPlatform.Samples/GeminiQuickstart.cs:line 82
   at GeminiQuickstartTest.TestGenerateContentAsync() in /Users/atamel/dev/github/meteatamel/dotnet-docs-samples/aiplatform/api/AIPlatform.Samples.Tests/GeminiQuickstartTest.cs:line 35
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|57_1()
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()

Once in a while, the sample works fine (maybe 1 out of 5 or so). There seems to be a problem between the GAPIC library and the Gemini 1.5 endpoint when streaming is being used.

I don’t know how to further debug this, so assigning back to you for recommendations.