cloud-sql-jdbc-socket-factory: "java.io.IOException: Error writing request body to server" error during create ephemeral certificate

Dear team,

I have been struggling for a month with the following issue. I have a Java 11 springboot project running from a Docker image based on the OpenJdk11 slim image (already tried various other base images and always got the same error) in a Cloud run environment conecting to google postgres SQL instance. The conection is done with the google socket factory and the datasource generated using hikari config with the SQL instance name (private acess, no public IP). The project runs fine for a few hours, sometime almost a few days and then it stops beeing able to connect to the SQL instance The first 2 errors are the following :

Error 1 java.lang.RuntimeException: [<Name-Of-Google-SQL-Instance>] Failed to create ephemeral certificate for the Cloud SQL instance. at com.google.cloud.sql.core.CloudSqlInstance.addExceptionContext (CloudSqlInstance.java:574) at com.google.cloud.sql.core.CloudSqlInstance.fetchEphemeralCertificate (CloudSqlInstance.java:515) at com.google.cloud.sql.core.CloudSqlInstance.lambda$performRefresh$0 (CloudSqlInstance.java:330) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:78) at java.util.concurrent.Executors$RunnableAdapter.call (Unknown Source) at java.util.concurrent.FutureTask.run (Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker (Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source) at java.lang.Thread.run (Unknown Source) Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at sun.security.ssl.SSLSocketImpl.handleEOF (Unknown Source) at sun.security.ssl.SSLSocketImpl.decode (Unknown Source) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord (Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake (Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake (Unknown Source) at sun.net.www.protocol.https.HttpsClient.afterConnect (Unknown Source) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0 (Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream (Unknown Source) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream (Unknown Source) at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:113) at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:84) at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1012) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:514) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:455) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute (AbstractGoogleClientRequest.java:565) at com.google.cloud.sql.core.CloudSqlInstance.fetchEphemeralCertificate (CloudSqlInstance.java:513) Error 2: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.SSLSocketInputRecord.read (Unknown Source) at sun.security.ssl.SSLSocketInputRecord.readHeader (Unknown Source) at sun.security.ssl.SSLSocketInputRecord.decode (Unknown Source) at sun.security.ssl.SSLTransport.decode (Unknown Source) at sun.security.ssl.SSLSocketImpl.decode (Unknown Source) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord (Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake (Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake (Unknown Source) at sun.net.www.protocol.https.HttpsClient.afterConnect (Unknown Source) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (Unknown Source) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect (Unknown Source) at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:148) at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:84) at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1012) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:514) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:455) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute (AbstractGoogleClientRequest.java:565) at com.google.cloud.sql.core.CloudSqlInstance.fetchMetadata (CloudSqlInstance.java:438) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:78) at java.util.concurrent.Executors$RunnableAdapter.call (Unknown Source) at java.util.concurrent.FutureTask.run (Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker (Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source) at java.lang.Thread.run (Unknown Source)

then at every attempt to connect to the project or when sheduled tasks run the following error :

java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.SSLSocketInputRecord.read (Unknown Source) at sun.security.ssl.SSLSocketInputRecord.readHeader (Unknown Source) at sun.security.ssl.SSLSocketInputRecord.decode (Unknown Source) at sun.security.ssl.SSLTransport.decode (Unknown Source) at sun.security.ssl.SSLSocketImpl.decode (Unknown Source) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord (Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake (Unknown Source) at sun.security.ssl.SSLSocketImpl.startHandshake (Unknown Source) at sun.net.www.protocol.https.HttpsClient.afterConnect (Unknown Source) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (Unknown Source) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect (Unknown Source) at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:148) at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:84) at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1012) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:514) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:455) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute (AbstractGoogleClientRequest.java:565) at com.google.cloud.sql.core.CloudSqlInstance.fetchMetadata (CloudSqlInstance.java:438) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:78) at java.util.concurrent.Executors$RunnableAdapter.call (Unknown Source) at java.util.concurrent.FutureTask.run (Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker (Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source) at java.lang.Thread.run (Unknown Source)

From what I understood it seems that this error appear when the ssl is renewd on the sql instance, and also understood that this is normal as far the conection is then retried and ends succesfull. (issue 310) In my case the conection is never re-established and the whole project stops working.

For the moment, as i am mostly doing trials to see how we could migrate our project to the Google cloud plateform, my current solution is to restart the google cloud run instance when this error occurs, but this is not a viable for a production solution… And idea where the error could be ? Thanks for your help to put me on right track !

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 74 (35 by maintainers)

Commits related to this issue

fix: add recommendation for Run and Functions Fixes #528 — committed to GoogleCloudPlatform/cloud-sql-jdbc-socket-factory by enocom 3 years ago

Most upvoted comments

The best solution in my opinion is to run the socket factory using a Unix socket. Note: in this approach you will need to create a Cloud SQL Connector.

Hi @enocom, I confirm, not a single error in 2 days working with unix socket and version 1.3.1 . Thanks all for your investigation and solution.

vbonne on Jul 25, 2021

After some digging and a number of conversations on our team, I’ve managed to reproduce this issue in a much smaller scale and have found the root of the problem.

In short, this is not a bug in the socket factory. Instead, it’s a result of running the socket factory on Cloud Run.

From the Cloud Run docs in the section Avoiding background activities:

When an application running on Cloud Run finishes handling a request, the container instance’s access to CPU will be disabled or severely limited. Therefore, you should not start background threads or routines that run outside the scope of the request handlers.

Running background threads can result in unexpected behavior because any subsequent request to the same container instance resumes any suspended background activity.

On a first request, the socket factory will initiate a connection to the Cloud SQL instance. It will also schedule a refresh of the associated credentials for that connection ~55 minutes in the future. What we’re seeing here is that refresh operation failing on account of being throttled by Cloud Run.

The best solution in my opinion is to run the socket factory using a Unix socket. Note: in this approach you will need to create a Cloud SQL Connector.

enocom on Jul 22, 2021

I see you’re using an entry for Cloud SQL Connectors, which will run the Cloud SQL proxy. In fact for Java you don’t need the Cloud SQL Connectors as the socket factory does all the work for you.

Try removing that and trying again.

restarted, will keep you posted

vbonne on Jul 2, 2021

I can’t say anything about appearing date of this error as my project is just been ported to GCP so I started with latest version. To answer your question I have just started the service with the version 1.2.0 of the postgres-socket-factory.

same error happened only 1h after startup :

java.lang.RuntimeException: [instance name] Failed to create ephemeral certificate for the Cloud SQL instance.
	at com.google.cloud.sql.core.CloudSqlInstance.addExceptionContext(CloudSqlInstance.java:465)
	at com.google.cloud.sql.core.CloudSqlInstance.fetchEphemeralCertificate(CloudSqlInstance.java:349)
	at com.google.cloud.sql.core.CloudSqlInstance.lambda$performRefresh$0(CloudSqlInstance.java:207)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Error writing request body to server
	at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(Unknown Source)
	at java.base/java.io.BufferedOutputStream.flushBuffer(Unknown Source)
	at java.base/java.io.BufferedOutputStream.flush(Unknown Source)
	at java.base/java.util.zip.DeflaterOutputStream.flush(Unknown Source)
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1193)
	at com.google.api.client.json.jackson2.JacksonGenerator.flush(JacksonGenerator.java:45)
	at com.google.api.client.http.json.JsonHttpContent.writeTo(JsonHttpContent.java:77)
	at com.google.api.client.http.GZipEncoding.encode(GZipEncoding.java:50)
	at com.google.api.client.http.HttpEncodingStreamingContent.writeTo(HttpEncodingStreamingContent.java:48)
	at com.google.api.client.http.javanet.NetHttpRequest$DefaultOutputWriter.write(NetHttpRequest.java:76)
	at com.google.api.client.http.javanet.NetHttpRequest.writeContentToOutputStream(NetHttpRequest.java:171)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:117)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
	at com.google.cloud.sql.core.CloudSqlInstance.fetchEphemeralCertificate(CloudSqlInstance.java:347)
	... 10 common frames omitted

but in this case after those errors were “normal” as the attemps were correctly retried and none of the scheduled tasks did actually fail as the conections managed to be re-established fast enough.

vbonne on Jun 29, 2021