moby: Docker 1.12.0-rc3 swarm mode - can't make wildfly standalon-ha to register on httpd-modcluster

Output of docker version:

docker version
Client:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   91e29e8
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   91e29e8
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 21
 Running: 2
 Paused: 0
 Stopped: 19
Images: 162
Server Version: 1.12.0-rc3
Storage Driver: devicemapper
 Pool Name: docker-202:32-3932161-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.345 GB
 Data Space Total: 107.4 GB
 Data Space Available: 102.9 GB
 Metadata Space Used: 8.495 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.139 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge host null
Swarm: active
 NodeID: 01h8iq4kxdlibdw9mdx394jd0
 IsManager: Yes
 Managers: 1
 Nodes: 3
 CACertHash: sha256:04669ba0d6dc01402b667248b5ac3c86319f55d07c34c09f1cbf46320f220b84
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.792 GiB
Name: VINDOC701.dev1.stelo.local
ID: E7SP:4FWC:HVFL:KIWG:3HIJ:ZXVZ:PYSE:4OZI:KGK7:J2LP:DK6F:XFSY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.): Physical

Steps to reproduce the issue: 1.

docker node ls
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
01h8iq4kxdlibdw9mdx394jd0 * VINDOC701.dev1.local Accepted Ready Active Leader
510vgjvdvq9kk5ufpzb44p7iu VINDOC703.dev1.local Accepted Ready Active 
7oq0krop57wfebcgsr8t8bk4t VINDOC702.dev1.local Accepted Ready Active
  1. I have two Docker Images I’ve built, one for httpd with mod_cluster, and another with Wildfly 10. When I run them on my local machine without Swarm, they work together fine, so when I query mod_cluster_manager, it shows all the Wildfly containers running, and it also load balance http access to node-info app:
dstolf@cherno-alpha:~/docker/wildfly$ docker run -d -p 8080:80 centos/httpd/modcluster
1b2de0f2c0d9867b39246d3cd22efb56676214099d494d6c4106a1a5b511961a

dstolf@cherno-alpha:~/docker/wildfly$ docker run -d jboss/wildfly/app
d60c191a7b72b17690310c6d41e0fb791d20372e54aaf0591f406d659511bb1f

dstolf@cherno-alpha:~/docker/wildfly$ docker run -d jboss/wildfly/app
5bb68830d6bc33210e384989cd8f1233fdaac54ef240c61fd11f2551d6052b4d

dstolf@cherno-alpha:~/docker/wildfly$ docker run -d jboss/wildfly/app
3f234b5174a03d1ff4c2b95d11e4f57a0b44ce479bbefc70e2a3bced03eb7b2f

dstolf@cherno-alpha:~/docker/wildfly$ curl http://localhost:8080/mod_cluster_manager
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html><head>
<title>Mod_cluster Status</title>
</head><body>
<h1>mod_cluster/1.3.1.Final</h1><a href="/mod_cluster_manager?nonce=bbe3e1d0-4053-47f1-9b4d-ded29e73874b&refresh=10">Auto Refresh</a> <a href="/mod_cluster_manager?nonce=bbe3e1d0-4053-47f1-9b4d-ded29e73874b&Cmd=DUMP&Range=ALL">show DUMP output</a> <a href="/mod_cluster_manager?nonce=bbe3e1d0-4053-47f1-9b4d-ded29e73874b&Cmd=INFO&Range=ALL">show INFO output</a>
<h1> Node server-172.17.0.3 (ajp://172.17.0.3:8009): </h1>
...
  1. So I pulled the originial images (jboss/wildfly and centos/httpd) from official registry into my private registry. When I run them on docker 1.12 with swarm mode active, the Wildfly containers can’t communicate with httpd-modcluster and register themselves:
docker service create --replicas 1  -p 3000:80    --mount target=/data,source=/data,type=bind,writable=true --name load_balancer dockerhub.dev1.stelo.local:5000/centos/httpd/modcluster
docker service create --replicas 3  -p 8080:8080   --mount target=/data,source=/data,type=bind,writable=true  --name java_app       dockerhub.dev1.stelo.local:5000/jboss/wildfly/app

4- So, when I go to my browser and try to open on :3000/mod_cluster_manager, there are no containers registered. If I go to :8080/node-info, I get the expected output from the test java app, because I published port 8080.

5- http/modcluster Dockerfile

FROM dockerhub.dev1.stelo.local:5000/centos/httpd
USER root

# Disable mod_proxy_balancer module and enable mod_cluster
RUN sed -i 's|LoadModule proxy_balancer_module|# LoadModule proxy_balancer_module|' /etc/httpd/conf.modules.d/00-proxy.conf 

# Add launch.sh 
ADD launch.sh /
ADD template_modcluster /
RUN chmod +x /launch.sh

# add mod_cluster 
ADD mod_advertise.so /etc/httpd/modules/
ADD mod_manager.so /etc/httpd/modules/
ADD mod_proxy_cluster.so /etc/httpd/modules/
ADD mod_cluster_slotmem.so /etc/httpd/modules/

# Add mod_cluster.conf 
ADD mod_cluster.conf /etc/httpd/conf.d/mod_cluster.conf

# Do the required modifications and launch Apache after boot
ENTRYPOINT /launch.sh

5- launch.sh

#/bin/bash

# Get the Hostname
HOSTNAME=$(hostname)
# Adjust the IP addresses in the mod_cluster.conf file
IPADDR=$(hostname -i | awk '{print $2}')
sed -i "s|[0-9\.\*]*:80|$IPADDR:80|g" /etc/httpd/conf.d/mod_cluster.conf

cat template_modcluster >> /etc/httpd/conf.d/mod_cluster.conf

# Log run
echo "######## Start httpd modcluster ########" >> /data/httpd/httpd_${HOSTNAME}.log
hostname >> /data/httpd/httpd_${HOSTNAME}.log
echo $IPADDR >> /data/httpd/httpd_${HOSTNAME}.log
date >> /data/httpd/httpd_${HOSTNAME}.log

# touch error_log and access_log
touch /var/log/httpd/error_log
touch /var/log/httpd/access_log

# tail file and redirect to /data/httpd/
nohup tail -f /var/log/httpd/error_log >> /data/httpd/error_log_${HOSTNAME}.log &
nohup tail -f /var/log/httpd/access_log >> /data/httpd/access_log_${HOSTNAME}.log &
nohup tail -f /var/log/messages >>  /data/httpd/messages_${HOSTNAME}.log &

# dump final mod_cluster.conf

5- template_modcluster

<VirtualHost *:80>

  EnableMCPMReceive true
  ServerAdvertise On
  ServerName loadbalancer

  <Location />
    Require all granted
  </Location>

  <Location /mod_cluster_manager>
    SetHandler mod_cluster-manager
    Require all granted
  </Location>

</VirtualHost>

6- wildfly docker file:

FROM dockerhub.dev1.stelo.local:5000/jboss/wildfly
USER root
# ADD app to deployment folder
ADD node-info.war /opt/jboss/wildfly/standalone/deployments/


# ADD launch script standalone ha mode
ADD launch_standalone_ha.sh /

# Change launch script permission
RUN chown jboss:jboss /launch_standalone_ha.sh

# Change app files permission
RUN chown jboss:jboss /opt/jboss/wildfly/standalone/deployments/node-info.war

# Change launch scripts permissions
RUN chmod +x /launch_standalone_ha.sh

# add wildfly admin user
USER jboss
RUN /opt/jboss/wildfly/bin/add-user.sh admin Admin@123 --silent

# Run WildFly after the container boots
ENTRYPOINT /launch_standalone_ha.sh

7- launch_standalone_ha.sh

#!/bin/bash
# output of hostname -I is 10.255.0.11 172.18.0.4 
# already tried to use both of them at the line bellow
IPADDR=$(hostname -I | awk '{print $2}')
touch /data/wildfly/standalone_ha_`hostname`.log
hostname -I > /data/wildfly/standalone_ha_`hostname`.log

# alternate launch #
#echo "/opt/jboss/wildfly/bin/standalone.sh -c standalone-ha.xml -Djboss.bind.address=$IPADDR -Djboss.bind.address.management=$IPADDR -Djboss.node.name=server-$IPADDR" >> /data/wildfly/standalone_ha_`hostname`.log

#/opt/jboss/wildfly/bin/standalone.sh -c standalone-ha.xml -Djboss.bind.address=$IPADDR -Djboss.bind.address.management=$IPADDR -Djboss.node.name=server-$IPADDR >> /data/wildfly/standalone_ha_`hostname`.log
# ############ #

echo "/opt/jboss/wildfly/bin/standalone.sh -c standalone-ha.xml -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=$IPADDR -Djboss.node.name=server-$IPADDR" >> /data/wildfly/standalone_ha_`hostname`.log
/opt/jboss/wildfly/bin/standalone.sh -c standalone-ha.xml -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=$IPADDR -Djboss.node.name=server-$IPADDR >> /data/wildfly/standalone_ha_`hostname`.log

Describe the results you received: Wildfly nodes don’t register themselves with http-modcluster

Describe the results you expected: That wildfly nodes should register with http-modcluster

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 17 (8 by maintainers)

Most upvoted comments

@thaJeztah https://github.com/docker/libnetwork/issues/552 and I added the help/wanted label long ago 😃

+1
mavenugo on Jul 14, 2016

@mavenugo do we have a tracking issue for that? Perhaps we should create one, and add a “help wanted” label

+1
thaJeztah on Jul 14, 2016

@DSTOLF with the built-in service discovery, one can easily get the ip-addresses of all the containers that backs a service. For example, if mod_cluster-enabled HTTPD Service is named mod-cluster-service, then a DNS query to tasks.mod-cluster-service will return all the ip-addresses of the containers across the cluster that backs this service. I guess one can use that to configure the unicast method.

Yes. Multicast is not supported yet and using the unicast fallback approach is the only way atm.

+1
mavenugo on Jul 13, 2016

+1 This is exactly what I also would like to achieve! Looks like Weave Net could help though I haven’t tried it yet: https://www.weave.works/guides/part-3-creating-and-scaling-multi-host-docker-deployment-with-swarm-and-compose-using-weave/

+1
bangnab on Jul 13, 2016
Read more comments on GitHub
← moby: Docker 1.12 RC4: Unable to start container with SELinux enabled on Oracle Linux 7.2
moby: Docker 1.13 unable to kill unresponsive containers  →