keycloak: [21.x] TCPPING: Unable to use any JGroups configuration mechanisms provided in properties

Before reporting an issue

  • I have searched existing issues
  • I have reproduced the issue with the latest release

Area

infinispan

Describe the bug

defining a custom transport for distributed cache is not recognized with:

  • keycloak 21.0.1/21.1.1,
  • openjdk-18-jre-headless: 18.0.2+9-2

in cache-ispn-tcpping.xml:

...
    <jgroups>
        <stack name="tcpping" extends="tcp">
          <TCP external_addr="${env.KC_EXTERNAL_ADDR}" bind_addr="${env.KC_BIND_ADDR}" bind_port="${env.KC_BIND_PORT}" />
          <TCPPING initial_hosts="${env.KC_INITIAL_HOSTS}"
                   max_dynamic_hosts="0"
                   port_range="0"
                   stack.combine="REPLACE"
                   stack.position="MPING"/>
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="tcpping"/>
...

In keycloak.conf:

...
cache=ispn
###cache-stack=tcp
...

Building with:

bin/kc.sh build --cache-config-file="cache-ispn-tcpping.xml"

and starting with:

KEYCLOAK_ADMIN=admin
KEYCLOAK_ADMIN_PASSWORD='my-password' 
KC_EXTERNAL_ADDR=172.16.1.1 
KC_BIND_ADDR=172.16.1.1 
KC_BIND_PORT=7800 
KC_INITIAL_HOSTS='"172.16.1.1[7800],172.16.1.2[7800]"'

bin/kc.sh start &
Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
Server configuration updated and persisted. Run the following command to review the configuration:

	kc.sh show-config

Next time you run the server, just run:

	kc.sh start --optimized 

ps -elf | grep -P "keycloak-[^/]+/bin/"
4 S keycloak 1961351 1961350  6  80   0 - 860363 futex_ 15:55 ?       00:00:36 java -Dkc.config.built=true -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.err.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED -Dkc.home.dir=/keycloak-21.1.1/bin/.. -Djboss.server.config.dir=/keycloak-21.1.1/bin/../conf -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dquarkus-log-max-startup-records=10000 -cp /keycloak-21.1.1/bin/../lib/quarkus-run.jar io.quarkus.bootstrap.runner.QuarkusEntryPoint start

leads to the log:

2023-05-16 16:09:21,969 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: kc1.example.com, Strict HTTPS: true, Path: <request>, Strict BackChannel: true, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: true
2023-05-16 16:09:25,390 INFO  [org.infinispan.SERVER] (keycloak-cache-init) ISPN005054: Native IOUring transport not available, using NIO instead: io.netty.incubator.channel.uring.IOUring
2023-05-16 16:09:25,534 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-05-16 16:09:25,572 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-05-16 16:09:25,606 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-05-16 16:09:26,422 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2023-05-16 16:09:26,570 WARN  [io.quarkus.vertx.http.runtime.VertxHttpRecorder] (main) The X-Forwarded-* and Forwarded headers will be considered when determining the proxy address. This configuration can cause a security issue as clients can forge requests and send a forwarded header that is not overwritten by the proxy. Please consider use one of these headers just to forward the proxy address in requests.
2023-05-16 16:09:26,579 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-05-16 16:09:26,593 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-05-16 16:09:26,598 INFO  [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 8fe49999-5dae-44d9-ac1f-e08a45de09f0, name: kc1-1500
2023-05-16 16:09:26,614 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 8.39MB
2023-05-16 16:09:26,619 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 8.39MB
2023-05-16 16:09:26,628 INFO  [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.34242
2023-05-16 16:09:28,644 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) kc1-1500: no members discovered after 2003 ms: creating cluster as coordinator
2023-05-16 16:09:28,652 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [kc1-1500|0] (1) [kc1-1500]
2023-05-16 16:09:28,656 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `kc1-1500`, physical addresses are `[172.16.1.1:49778]`
2023-05-16 16:09:29,231 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: kc1-1500, Site name: null
2023-05-16 16:09:31,071 INFO  [io.quarkus] (main) Keycloak 21.1.1 on JVM (powered by Quarkus 2.13.7.Final) started in 11.321s. Listening on: http://127.0.0.1:80 and https://127.0.0.1:443
2023-05-16 16:09:31,072 INFO  [io.quarkus] (main) Profile prod activated. 
2023-05-16 16:09:31,072 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, vertx]
2023-05-16 16:09:31,104 ERROR [org.keycloak.services] (main) KC-SERVICES0010: Failed to add user 'admin' to realm 'master': user with username exists

Version

21.1.1

Expected behavior

  • accepted custom transport
  • discovered members

Actual behavior

Unable to use any JGroups configuration mechanisms provided in properties

How to Reproduce?

Already explained above

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

@jean-christophe-manciot Thank you for reporting the issue. I think this is problem with configuration.

I tried to unzip 2 keycloak-21.1.1 distributions, configured cache-ispn.xml accordingly:

...
<jgroups>
   <stack name="tcpping" extends="tcp">
    <TCP bind_port="7800" /> // or 7801 based on the instance
    <TCPPING
          initial_hosts="127.0.0.1[7800],127.0.0.1[7801]"
          max_dynamic_hosts="0"
          port_range="0"
          stack.combine="REPLACE"
          stack.position="MPING"
         />
     </stack>
</jgroups>
<cache-container name="keycloak">
    <transport lock-timeout="60000" stack="tcpping"/>
...

and then start each instance like:

./bin/kc.sh start-dev --cache=ispn --cache-stack=tcp --cache-config-file=cache-ispn.xml --http-port=8080

./bin/kc.sh start-dev --cache=ispn --cache-stack=tcp --cache-config-file=cache-ispn.xml --http-port=8081

then I see in the log

ISPN100010: Finished rebalance with members [localhost-47707, localhost-13796], topology id 2

Try to provide command line options --cache=ispn --cache-stack=tcp. You might need to run ./bin/kc.sh build to make any change in your configuration effective. As you can see, I tried this on my local machine, so you might want to double-check, there is no network problem on your side.

Closing this as explained, but feel free to reopen if you still think there is a bug.

+2
martin-kanis on May 23, 2023

@martin-kanis @jean-christophe-manciot I think the confusion is around the usage of the optimized option.

After running the build command you must pass --optimized in order to avoid another re-build as per:

Next time you run the server, just run:

	kc.sh start --optimized

Without this option, the server is going to revert the changes you did when running build and use the default UDP stack accordingly.

That is why @martin-kanis is facing a different behavior. He is passing everything within a single command (e.g.: start-dev or start) and in this case you are going to have a build being executed behind the scenes.

For more details, see https://www.keycloak.org/server/configuration#_optimize_the_keycloak_startup.

It is not really your fault, I hope we can make this more clear in our docs …

+1
pedroigor on May 24, 2023
Read more comments on GitHub
← keycloak: 18.1 conf/keycloak.conf can't process https-certificate-file
keycloak: 403 login-status-iframe on Keycloak Docker Compose →