docker-plugin: Can't make plugin to work over unix socket

Jenkins and plugins versions report

Jenkins: 2.433
OS: Linux - 4.15.0-135-generic
Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:11.13.0
ansicolor:1.0.4
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.2.2-1.0
authentication-tokens:1.53.v1c90fd9191a_b_
badge:1.9.1
blueocean:1.27.9
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.9
blueocean-commons:1.27.9
blueocean-config:1.27.9
blueocean-core-js:1.27.9
blueocean-dashboard:1.27.9
blueocean-display-url:2.4.2
blueocean-events:1.27.9
blueocean-git-pipeline:1.27.9
blueocean-github-pipeline:1.27.9
blueocean-i18n:1.27.9
blueocean-jwt:1.27.9
blueocean-personalization:1.27.9
blueocean-pipeline-api-impl:1.27.9
blueocean-pipeline-editor:1.27.9
blueocean-pipeline-scm-api:1.27.9
blueocean-rest:1.27.9
blueocean-rest-impl:1.27.9
blueocean-web:1.27.9
bootstrap5-api:5.3.2-2
bouncycastle-api:2.29
branch-api:2.1135.v8de8e7899051
build-name-setter:2.4.0
build-timeout:1.31
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloud-stats:320.v96b_65297a_4b_b_
cloudbees-bitbucket-branch-source:856.v04c46c86f911
cloudbees-disk-usage-simple:187.v6378d330d1d4
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-94.v3e1f4a_926e49
conditional-buildstep:1.4.3
config-file-provider:959.vcff671a_4518b_
configuration-as-code:1746.vf1673cfe690a
convert-to-pipeline:1.0
copyartifact:722.v0662a_9b_e22a_c
credentials:1309.v8835d63eb_d8a_
credentials-binding:642.v737c34dea_6c2
dashboard-view:2.495.v07e81500c3f2
data-tables-api:1.13.8-1
delivery-pipeline-plugin:1.4.2
description-setter:1.10
disk-usage:1.2
display-url-api:2.200.vb_9327d658781
docker-build-step:2.10
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.1-79.v20b_53427e041
docker-plugin:1.3.0
docker-slaves:1.0.7
docker-workflow:572.v950f58993843
downstream-build-cache:1.7
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.3-1
extended-choice-parameter:376.v2e02857547b_a_
external-monitor-job:215.v2e88e894db_f8
extra-tool-installers:113.v5d95d475e985
favorite:2.4.3
font-awesome-api:6.4.2-1
forensics-api:2.3.0
git:5.2.1
git-client:4.5.0
git-server:99.va_0826a_b_cdfa_d
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1751.v90e17c48a_6a_c
gitlab-plugin:1.7.16
global-variable-string-parameter:1.2
google-login:1.8
google-metadata-plugin:0.5
google-oauth-plugin:1.318.vb_39c5db_e3041
google-play-android-publisher:4.2
google-source-plugin:0.4
gradle:2.9
h2-api:11.1.4.199-12.v9f4244395f7a_
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
hetzner-cloud:84.v8acf5510fd35
htmlpublisher:1.32
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.3-372.v309620682326
jacoco:3.3.5
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.9
jersey2-api:2.41-133.va_03323b_a_1396
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.87
jobConfigHistory:1229.v3039470161a_d
join:1.21
jquery:1.12.4-1
jquery3-api:3.7.1-1
jsch:0.2.8-65.v052c39de79b_2
junit:1240.vf9529b_881428
kubernetes-cli:1.12.1
kubernetes-client-api:6.8.1-224.vd388fca_4db_3b_
kubernetes-credentials:0.11
mailer:463.vedf8358e006b_
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.2.1
matrix-project:818.v7eb_e657db_924
maven-plugin:3.23
metrics:4.2.18-442.v02e107157925
mina-sshd-api-common:2.11.0-86.v836f585d47fa_
mina-sshd-api-core:2.11.0-86.v836f585d47fa_
naginator:1.421.v85c11e2db_507
next-executions:239.v27031b_e26418
oauth-credentials:0.646.v02b_66dc03d2e
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
parameterized-trigger:787.v665fcf2a_830b_
performance:951.v5600a_c6422ed
perfpublisher:8.09
pipeline-build-step:516.v8ee60a_81c5b_9
pipeline-config-history:1.6
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-maven:1362.vee39a_d4b_02b_1
pipeline-maven-api:1362.vee39a_d4b_02b_1
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2151.ve32c9d209a_3f
pipeline-model-definition:2.2151.ve32c9d209a_3f
pipeline-model-extensions:2.2151.ve32c9d209a_3f
pipeline-npm:155.ve2e393062d86
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f
pipeline-stage-view:2.34
pipeline-timeline:1.0.3
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.6.0
prism-api:1.29.0-9
pubsub-light:1.18
rebuild:330.v645b_7df10e2a_
resource-disposer:0.23
rich-text-publisher-plugin:1.5
robot:3.4.0
role-strategy:689.v731678c3e0eb_
run-condition:1.7
scm-api:683.vb_16722fb_b_80b_
script-security:1281.v22fb_899df1a_e
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.16.1
sse-gateway:1.26
ssh:2.6.1
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
sshd:3.312.v1c601b_c83b_0e
structs:325.vcb_307d2a_2782
timestamper:1.26
token-macro:384.vf35b_f26814ec
toolenv:1.2
trilead-api:2.84.v72119de229b_7
variant:60.v7290fc0eb_b_cd
view-job-filters:369.ve0513a_a_f5524
warnings-ng:10.5.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3806.va_3a_6988277b_2
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1360.vc6700e3136f5
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45
yet-another-build-visualizer:1.16

What Operating System are you using (both controller, and any agents involved in the problem)?

OS: Ubuntu LTS Docker: 20.10.7

Reproduction steps

run jenkins controller using docker-compose:

---
version: '3.2'
services:
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins:lts
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        tag: "{{.Name}}"
    ports:
    - 8000:8000
    - 8084:8080
    - 50000:50000
    volumes:
    - /usr/bin/docker:/usr/bin/docker:ro
    - /var/run/docker.sock:/var/run/docker.sock:rw
    - /opt/jenkins/data:/var/jenkins_home

Configure docker-local cloud using script from this repo

Expected Results

Docker plugin to connect to docker daemon

Actual Results

Error message that plugin can’t connect: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

Anything else?

Here are detailed logs

Nov 27, 2023 10:10:26 AM FINE com.nirima.jenkins.plugins.docker.DockerContainerWatchdog execute
Checking Docker Cloud docker-local at unix:///var/run/docker.sock
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalHttpClient doExecute
ex-0000000001 preparing request execution
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ProtocolExec execute
ex-0000000001 target auth state: UNCHALLENGED
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ProtocolExec execute
ex-0000000001 proxy auth state: UNCHALLENGED
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ConnectExec execute
ex-0000000001 acquiring connection with route {}->unix://localhost:2375
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime acquireEndpoint
ex-0000000001 acquiring endpoint (3 MINUTES)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager lease
ex-0000000001 endpoint lease request (3 MINUTES) [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$3 get
ex-0000000001 endpoint leased [route: {}->unix://localhost:2375][total available: 0; route allocated: 1 of 2147483647; total allocated: 1 of 2147483647]
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$3 get
ex-0000000001 acquired ep-0000000001
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime acquireEndpoint
ex-0000000001 acquired endpoint ep-0000000001
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ConnectExec execute
ex-0000000001 opening connection {}->unix://localhost:2375
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime connectEndpoint
ep-0000000001 connecting endpoint (3000000000 NANOSECONDS)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager connect
ep-0000000001 connecting endpoint to unix://localhost:2375 (3000000000 NANOSECONDS)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost resolving remote address
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost resolved to [localhost/127.0.0.1]
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost:2375 connecting null->localhost/127.0.0.1:2375 (3000000000 NANOSECONDS)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost:2375 connection to localhost/127.0.0.1:2375 failed (class java.net.ConnectException); terminating operation
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection close
http-outgoing-0 close connection IMMEDIATE
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime discardEndpoint
ep-0000000001 endpoint closed
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime discardEndpoint
ep-0000000001 discarding endpoint
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager release
ep-0000000001 releasing endpoint
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager release
ep-0000000001 connection is not kept alive
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager release
ep-0000000001 connection released [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
Nov 27, 2023 10:10:26 AM WARNING com.nirima.jenkins.plugins.docker.DockerContainerWatchdog retrieveContainers
Unable to retrieve list of containers available on DockerCloud [name=docker-local, dockerURI=unix:///var/run/docker.sock] while reading list of containers (showAll=true, labelFilters={com.nirima.jenkins.plugins.docker.JenkinsId=8de6877e67724162b1587711f7c0950b})
org.apache.hc.client5.http.HttpHostConnectException: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.lambda$connectSocket$0(PlainConnectionSocketFactory.java:91)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:90)
	at org.apache.hc.client5.http.socket.ConnectionSocketFactory.connectSocket(ConnectionSocketFactory.java:123)
	at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:184)
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:191)
Caused: java.lang.RuntimeException
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195)
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
	at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
	at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202)
	at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:74)
	at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:44)
	at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:15)
	at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
	at com.nirima.jenkins.plugins.docker.DockerContainerWatchdog.retrieveContainers(DockerContainerWatchdog.java:263)
	at com.nirima.jenkins.plugins.docker.DockerContainerWatchdog.processCloud(DockerContainerWatchdog.java:208)
	at com.nirima.jenkins.plugins.docker.DockerContainerWatchdog.execute(DockerContainerWatchdog.java:148)
	at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:102)
	at java.base/java.lang.Thread.run(Thread.java:829)

Nov 27, 2023 10:10:26 AM INFO com.nirima.jenkins.plugins.docker.DockerContainerWatchdog execute
Not checking the list of nodes, as list of containers is known to be incomplete

I tried to connection from jenkins master to socket using curl, no issue:

curl  -s --unix-socket /var/run/docker.sock http://v1/containers/json

[{"Id":"a53f9b6632cfc9d3132af8beb6adf1b8477609970411884fce4327b31a254a6e","Names":["/jenkins"],"Image":"docker.ifne.eu/ci-master","ImageID":"sha256:477287f0c313d59508d47028a3b7c5c078cc0013bb1ddd72fbfaae1a0b781288","Command":"/usr/bin/tini -- /usr/local/bin/jenkins.sh","Created":1701068662,"Ports":[{"IP":"0.0.0.0","PrivatePort":50000,"PublicPort":50000,"Type":"tcp"},...

I was also able to attach strace and trace down actual attempt for connection:

[pid 26737] socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 801
[pid 26737] setsockopt(801, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
[pid 26737] setsockopt(801, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 26737] setsockopt(801, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0
[pid 26737] fcntl(801, F_GETFL <unfinished ...>
[pid 26737] connect(801, {sa_family=AF_INET, sin_port=htons(2375), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)

Clearly, there should be AF_UNIX, but there is AF_INET.

Are you interested in contributing a fix?

Sure, happy to help if you give me pointers.