opentelemetry-java-instrumentation: Running JDBC instrumentation causing delays in socket read from DB

Describe the bug Running JDBC instrumentation causing large running times for threads wanting to socket read from DB . This causes application threads wanting to read from DB to taken longer , causing thread pool (Tomcat) starvation. This appears to be some sort of overhead introduced by the JDBC instrumentation running.

Steps to reproduce Use the JDBC instrumentation in Java OpenTelemery agent and observe load on DB CPU and also Write IOPs and application thread running times

What did you expect to see? The service runs as normal without any potential issues causing DB CPU to go up more than expected, or application threads (Tomcat) to start waiting more than often on socket read from DB.

What did you see instead? The service started getting longer than running threads that read from DB, causing threadpool starvation in the web service.

What version and what artifacts are you using? Artifacts: (opentelemetry-bom, opentelemetry-javaagent, opentelemetry-sdk , opentelemetry-api, zipkin exporter) Version: (v1.21.0) How did you reference these artifacts? (pom.xml)

Environment Compiler: “Java 11 temurin11” OS: “Debian 11.6” Runtime (if different from JDK above): OS (if different from OS compiled on):

Additional context To confirm this, we found that the issue went away once we turned the JDBC instrumentation off. The name of the instrumentation is otel.instrumentation.jdbc.enabled which is set to false when running the JAR. This can also be confirmed here: https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/#suppressing-specific-agent-instrumentation

About this issue

Original URL
State: open
Created a year ago
Comments: 15 (6 by maintainers)

Most upvoted comments

Library instrumentation classes are shaded under a different package name in agent to avoid conflicts when application bundles the same library. In intellij you can set a method breakpoint based on the class name, you can use the shaded name there and don’t need to have the source code for the class. It is possible to build the agent without shading, see https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/contributing/debugging.md#shadow-renaming

laurit on Apr 28, 2023

So you have implemented a wrapper for java.sql.Connection but you don’t implement java.sql.Wrapper? Agent uses

if (connection.isWrapperFor(Connection.class)) {
  connection = connection.unwrap(Connection.class);
}

to unwrap the connection and caches JdbcUtils.computeInfo based on the unwrapped connection. If the unwrapping fails agent will execute JdbcUtils.computeInfo for each wrapped connection. You could try implementing the wrapper interface so that agent could get the underlying connection and see if this improves the situation.

laurit on Apr 18, 2023