keycloak: adding package using microdnf no longer works since keycloak version 21

You can easily find this out by checking the changes in the layers using for example: https://github.com/wagoodman/dive

#16879 is IMO a mistake. This forces users that want to extend the the base image to do super brittle reverse engineered custom images just because they want to augment the system in any small way.

The workaround by @tobhv looks brittle for any changes keycloak will do in the future and already has issues

Initial testing looks fine with this for new installations, but not for upgrades

In the workaround by @3XC1T3D they are already worried that they may have or may not have bulldozed ENV variables and managed to forget the jdk

I can understand why one would lock down the image due to security concerns but I consider this to be overall worse than before.

As context what we do in our dockerfile

• copy another CA into the image and do a update-ca-trust which fails because the underlying utility p11-kit is now gone (probably will have to use a truststore instead from now on)
• and a microdnf install iproute to do some docker swarm required introspection.

FROM quay.io/keycloak/keycloak:21.0.0 as builder

ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENV KC_DB=postgres
ENV KC_CACHE_CONFIG_FILE=cache-ispn-jdbc-ping.xml
COPY deployments/. /opt/keycloak/providers/
COPY conf/. /opt/keycloak/conf/
RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:21.0.0
COPY domentry-net-ca.crt /etc/pki/ca-trust/source/anchors/domentry-net-ca.crt
USER root
RUN chmod 644 /etc/pki/ca-trust/source/anchors/domentry-net-ca.crt \
&& update-ca-trust 
RUN microdnf update -y && microdnf install iproute
COPY custom-entry.sh /opt/keycloak/bin/custom-entry.sh
RUN chmod 777 /opt/keycloak/bin/custom-entry.sh
RUN chown keycloak /opt/keycloak/bin/custom-entry.sh
USER 1000

COPY themes/. /opt/keycloak/themes/
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
WORKDIR /opt/keycloak
COPY deployments/. ./providers/ 
COPY conf/. ./conf/
HEALTHCHECK --start-period=120s CMD curl --fail http://localhost:8080/health || exit 1

ENTRYPOINT ["/opt/keycloak/bin/custom-entry.sh", "start", "--optimized"]

we are issung the same problem. it is not possible to offer two images? one based on ubi9-minimal and the other based on ubi9-micro?

So that the community dont have to rebuild each Dockerfile or CI/CD step.

HTH, I use another workaround - checking when 8080 port is reserved on the container. It is usually just before container to became healthy.

The healthcheck section is like this:

    healthcheck:
      test: cat /proc/net/tcp | grep '00000000:1F90 00000000:0000' || exit 1
      interval: 5s
      timeout: 2s
      retries: 20
      start_period: 120s

Explanation is that in /proc/net/tcp can be found a list of all ports that are used on the system (for tcp protocol), but in HEX format. 00000000:1F90 can be read as local address: 0.0.0.0 on port: 1F90 (equal to 8080 in dec)

So the check will pass when some process start to use this port inside the container.

Going back to 20, let me know when documentation is up-to-date, just need curl for healthcheck, wasted too much time…

My workaround for missing curl is to add static curl:

FROM quay.io/keycloak/keycloak:latest

# see https://github.com/aerogear/keycloak-metrics-spi/releases
ARG KEYCLOAK_METRICS_SPI_RELEASE=2.5.3

# Keycloak 21+ doesn't have curl, workaround for https://github.com/keycloak/keycloak/issues/17273
ADD --chown=root:root https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 /usr/bin/curl
USER root
RUN chmod +x /usr/bin/curl

USER keycloak
RUN \
  curl -L -k https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar \
  -o /tmp/opentelemetry-javaagent.jar && \
  curl -L -k https://github.com/aerogear/keycloak-metrics-spi/releases/download/${KEYCLOAK_METRICS_SPI_RELEASE}/keycloak-metrics-spi-${KEYCLOAK_METRICS_SPI_RELEASE}.jar \
  -o /opt/keycloak/providers/keycloak-metrics-spi.jar

I would recommend this workaround only if you understand security risks of this approach.

I don’t like an idea of full image build - it will introduces Docker image management problems. Also this is not a for prod deployment, so I’m happy to accept security risks of this workaround.

@bruegth Wait, can you explain how that works? monocle_face

It’s using the pseudo-device /dev/tcp (which is a Bash feature) to perform a TCP connection, see https://tldp.org/LDP/abs/html/devref1.html

3<> is opening a read/write file descriptor for the socket communication, see https://tldp.org/LDP/abs/html/io-redirection.html

Finally, grep is used to check if the strings “status” and “UP” appears in the response to exit the script successfully or with a none-zero exit code in case of an error.

@vmuzikar The it’s only kubectl

I tried to copy the export into the new 21.02 container kubectl cp keycloak-export.json thens/keycloak-754d88f645-54ln7:/tmp/keycloak-export.json

Result:

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "45f7be...": OCI runtime exec failed: exec failed: unable to start container process: exec: "tar": executable file not found in $PATH: unknown

I resolved it by creating the export as a configMap and mount it under /opt/keycloak/data/import/keycloak-export.json and appended the start arg --import-realm

Based on https://ceh51.blogspot.com/2016/07/how-to-open-tcpudp-sockets-bash-shell.html a docker-healthcheck.sh:

#!/bin/bash
exec 3<>/dev/tcp/localhost/8080

echo -e "GET /auth/health/ready HTTP/1.1\nhost: localhost:8080\n" >&3

timeout --preserve-status 1 cat <&3 | grep -m 1 status | grep -m 1 UP
ERROR=$?

exec 3<&-
exec 3>&-

exit $ERROR

What’s a real world attack surface added by curl? I mean, even /dev/tcp still exists and can be used for reserve shells and file transfers while for users it’s now pretty hard to get native healthcheck support working again

@xoxys yes, it was hard … 😉

As an alternative to a dynamically linked curl i tried a statically linked binary of xh ( https://github.com/ducaale/xh )

My Dockerfile:

FROM quay.io/keycloak/keycloak:21.0.0 as builder

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=scripts
ENV KC_DB=postgres
ENV KC_HTTP_RELATIVE_PATH=/auth

WORKDIR /opt/keycloak

RUN /opt/keycloak/bin/kc.sh build

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN export XH_BINDIR=/usr/local/bin && curl -sfL https://raw.githubusercontent.com/ducaale/xh/master/install.sh | sh

FROM quay.io/keycloak/keycloak:21.0.0
COPY --from=builder /opt/keycloak/ /opt/keycloak/
COPY --from=ubi-micro-build /usr/local/bin/xh /usr/local/bin/xh
WORKDIR /opt/keycloak
COPY theme/sinno-prod/ /opt/keycloak/themes/sinno-prod/
COPY theme/sinno-test/ /opt/keycloak/themes/sinno-test/

In my docker-compose.yml i now have a healthcheck section:

        healthcheck:
          test: ["CMD", "/usr/local/bin/xh", " --check-status", "http://127.0.0.1:8080/auth/health/ready"]

I moved to using ubi9-minimal as a base using the below dockerfile:

ARG QUAY_REGISTRY
ARG REGISTRY
ARG KC_VER
FROM ${QUAY_REGISTRY}/keycloak/keycloak:${KC_VER} as builder
# support meterics and health check
ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
# configure mariadb as the db to be supported
ENV KC_DB=mariadb
# support secrets
ENV KC_VAULT=file
RUN /opt/keycloak/bin/kc.sh build

#use redhat image that supports installing packages (i use curl, jq for the healthcheck)
FROM ${REGISTRY}/redhat/ubi9-minimal

COPY --from=builder /opt/keycloak/ /opt/keycloak/
COPY rootfs /
USER root
RUN microdnf install -y --nodocs --setopt=install_weak_deps=0 \
    java-17-openjdk-headless \
    glibc-langpack-en \
    jq \
    shadow-utils;\
    chmod +x /usr/local/bin/health.sh; \
    useradd -s /sbin/nologin keycloak; \
    microdnf remove libsemanage shadow-utils -y; \
    microdnf clean all;\
    rm -rf \
    /var/lib/dnf
USER keycloak
HEALTHCHECK CMD /usr/local/bin/health.sh
WORKDIR /opt/keycloak
ENV KC_METRICS_ENABLED=true
ENV KC_HEALTH_ENABLED=true
ENV LANG en_US.UTF-8
ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]

Initial testing looks fine with this for new installations, but not for upgrades in my case but that is due to https://github.com/keycloak/keycloak/issues/17285 Reason to use the ubi9-minimal from now on is that I want the option to install packages @runtime in case of troubleshooting.

@Zetanova Thank you for providing more details. I can confirm that kubectl cp indeed doesn’t work due to missing tar in the container.

Apart from the mentioned ConfigMap workaround, another possible solution might be using something as simple as:

kubectl exec example-kc-0 -- cat /opt/keycloak/data/import/keycloak-export.json > keycloak-export2.json

Could you please confirm this works for your use case? I guess it depends on how big your Realm is. 😃

@ghenadiibatalski Yes i did it in the end. I updated my last message 1min later.

I tested now kubectl cp thens/keycloak-754d88f645-54ln7:/opt/keycloak/data/import/keycloak-export.json keycloak-export2.json

Result:

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "823800...": OCI runtime exec failed: exec failed: unable to start container process: exec: "tar": executable file not found in $PATH: unknown

@t-schuster Your use cases are covered, as detailed in previous comments in this thread.

1. Do your certificate work in one stage FROM ubi9, where all the packages you need are available, then copy across /etc/pki into the second stage.
2. To install custom provider JARs, just use ADD --chown=keycloak:keycloak. You don’t need curl to access URLs.

Both of these use cases are now included in the latest container documentation, but we’re still working on the backport to the KC 21 docs.

To avoid opening a duplicate:

The issue prevents installing custom certificate-authorities reliably into the container and in our build pipeline also stops usage of curl/wget to fetch custom providers in the Dockerfile to provide them for the build.

Allowing p11-kit to be installed would help a lot here instead of having to create a brittle and fragile script to manually put the certificate stores into proper order.

I also stumbled across this method, but at least for me, it did not work reliably, as only unencrypted connections are supported. Depending on the KC_PROXY configuration, port 8080 may or may not work. Personally, I am considering using an approach similar to this https://github.com/parkr/go-curl

Then it might help if a java way to execute healthckecks would exist e.g. java run-health.jar.

We were actually considering this but realized that would probably hurt the performance – creating a new JVM for each health check request. Hence we see installing curl as a lesser evil.

I think removing essential stuff like a package manager is not the best solution when your base idea is to build an individual “own” image.

In the past I just copied our ActiveDirectory root CA into the anchor directory and update-ca-trust. update-ca-trust updates also the java default truststore.

update-ca trust is missing p11-kit -> The option to install it is also gone.

IMHO: This makes everything overcomplicated.

Now we can fight around with keytool instead of the build in tools of RH. And i think also copiing stuff manually for whatever needs to be installed like: executables, libraries etc. will make it even worse

Hi @ASzc ,

thanks that you take a look at the docs and update it.

1.) i think no one wants a static curl binary link in his Dockerfile. The major people wants the package from the based repository to acchive the latest security fixes etc.

2.) in this example you have to know which folders you have to copy from one layer to the other. In the example from me and @tobhv you dont have to worry about the folders to copy.

that’s my opinion 😃

This is due to https://github.com/keycloak/keycloak/pull/16879 and an intended change to minimise image size and attack surface. You can still install your own dependencies by using a multi-layer build i.e. create your own layer with microdnf, install your dependencies, switch to the Keycloak image and copy your installed dependencies from your own layer.