quarkus: Problem with environment variables when using multiple datasources

Describe the bug I belive there’s a problem with environment variables when using multiple datasources. Looks like Quarkus isn’t recognizing the environment variables that have datasource namespaces, like quarkus.datasource."users".jdbc.url . Or perhaps I’m not passing the variables correctly.

Expected behavior Quarkus should read the environment variables with the datasource namespace.

Actual behavior Quarkus isn’t reading reading the environment variables with the datasource namespace.

To Reproduce

  1. Build the project with mvn clean package
  2. Export the environment variables
  3. Execute the runnable jar.
  4. Quarkus says that the variables weren’t defined.

Configuration

quarkus.datasource."users".db-kind=postgresql
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.hibernate-orm."users".datasource=users
quarkus.hibernate-orm."users".packages=com.helesto.models.users
quarkus.hibernate-orm."users".database.generation = none
quarkus.hibernate-orm."users".sql-load-script=no-file
quarkus.hibernate-orm."users".log.sql=false

quarkus.datasource."orders".db-kind=mariadb
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}
quarkus.hibernate-orm."orders".datasource=orders
quarkus.hibernate-orm."orders".packages=com.helesto.models.orders
quarkus.hibernate-orm."orders".database.generation=none
quarkus.hibernate-orm."orders".sql-load-script=no-file
quarkus.hibernate-orm."orders".log.sql=false

Environment:

  • uname -a: Linux machine 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

  • java -version: openjdk version “11.0.8” 2020-07-14 OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

  • Quarkus version or git rev: 1.9.0.Final

  • mvnw --version: Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /home/helesto/.m2/wrapper/dists/apache-maven-3.6.3-bin/1iopthnavndlasol9gbrbg6bf2/apache-maven-3.6.3 Java version: 11.0.8, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64 Default locale: en_US, platform encoding: UTF-8 OS name: “linux”, version: “5.4.0-52-generic”, arch: “amd64”, family: “unix”

Test with my project - happy end

It’s possible to reproduce this problem with a project I created to test the use of multiple databases:

With this project, I did the following test:

  1. Build it with ./mvnw clean package
  2. Start the docker containers of the databases (PostgreSQL and MariaDB):
docker run -d --name postgres-db -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=usersdb postgres
docker run -d --name mariadb-db -p 3306:3306 -e MYSQL_USER=maria -e MYSQL_ROOT_PASSWORD=maria -e MYSQL_PASSWORD=maria -e MYSQL_DATABASE=ordersdb mariadb
  1. Run the project in dev mode just to create the tables and insert some rows:
./mvnw compile quarkus:dev
  1. Enter inside the /target folder and run the runnable jar passing all parameters:
java -jar -Dquarkus.datasource.users.jdbc.url=jdbc:postgresql://localhost:5432/usersdb -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar
  1. Enter inside the http://localhost:8080/swagger-ui , execute the tests and confirm everything is ok.

Now the problem.

  1. I try to export just one variable with, like this one:
export QUARKUS_DATASOURCE_USERS_JDBC_URL=jdbc:postgresql://localhost:5432/usersdb 
  1. Then I try to run the same command without this property ( quarkus.datasource."users".jdbc.url ), just to test:
java -jar -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar
  1. Quarkus ignore my environment variable and I receive this error:
WARN  [io.agr.pool] (main) Datasource 'users': Driver does not support the provided URL: prod-url
  • prod-url is my default value:
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}

Now my workaround

As Quarkus isn’t reading the property variables with the datasource namespace, I’ve created the variables below:

quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}

For example:

  1. I export the POSTGRESQL_URL:
export POSTGRESQL_URL=jdbc:postgresql://localhost:5432/usersdb 
  1. Now I run the application without passing this variable ( quarkus.datasource."users".jdbc.url ) and everything works fine:
java -jar -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar

I’m using this solution to run my real application inside Kubernetes. That is, instead of passing QUARKUS_DATASOURCE_USERS_JDBC_URL and the other variables, I’m passing this variables I’ve created, like POSTGRESQL_URL.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (17 by maintainers)

Most upvoted comments

After some examination - and this issue - I come to the conclusion that it’s only possible to overwrite namespaced properties that already exist in application.properties via environment variables but not to introduce them.

Correct. We need the property defined in their dotted format, so we can disambiguate when the property is set in the environment variable name.

I’m having basically the same problem with namespaced property quarkus.vault.credentials-provider."credentials-provider".database-credentials-role in a Kubernetes environment using Vault database engine for dynamic credentials.

My idea was to not include it in application.properties at all but instead pass it only via Kubernetes ConfigMap as envFrom block. I used the following in my ConfigMap:

QUARKUS_VAULT_CREDENTIALS_PROVIDER_MYDATABASE_DATABASE_CREDENTIALS_ROLE: my-role

…but I always end up with

io.quarkus.vault.VaultException: unknown credentials provider with name mydatabase

It all works perfectly if I put

quarkus.vault.credentials-provider.mydatabase.database-credentials-role = my-role

to application.properties but I would rather prefer not to have it there (keep application.properties as empty as possible).

After some examination - and this issue - I come to the conclusion that it’s only possible to overwrite namespaced properties that already exist in application.properties via environment variables but not to introduce them.

Can anyone confirm?

Great! Thanks!

In Quarkus we do canonicalize (to a degree) the quoting of property names, so foo\.bar should be equivalent to "foo.bar", and foo should be equivalent to "foo". If we’re seeing behavior that contradicts this, then I’d consider it a bug in the properties mapping.

On the topic of environment variable names: it’s quite tricky because the mapping only goes one way. Your description is therefore slightly inaccurate: we do not convert FOO_BAR to foo.bar but we do convert foo.bar to FOO_BAR when we probe the environment to see what properties are present. The conversion cannot be two-way because converting to environment vars is a lossy operation. For example we would consider foo.bar_baz and foo_bar.baz to be distinct configuration property names, but both of these would be mapped to an environment variable named FOO_BAR_BAZ.

@radcortez I was able to override a quoted property passing an unquoted -D argument to the java -jar execution. But when exporting the same unquoted property (with export var = ... ) and then executing the jar, it didn’t work.

Weird. As far as I know, we don’t treat quoted property names differently, in the sense that the quote is part of the name, meaning that you require the quote in the system property name to be able to override it. I’ve tried in one of my projects and I was only able to override it when using the quoted version.

The issue with environment variables is obvious, since you are not able to use quotes on their name it renders any override useful for quote property names. Also an environment variable name is just converted to its property name counter part, meaning that something like FOO_BAR gets converted to foo.bar to perform the lookup.

@dmlloyd do you think we should convert quoted property names to their unquoted counterparts transparently? Right now, they are just treated as different properties.

Hum… can’t remember anything related, but I’m going to investigate.

@radcortez does it ring a bell?

I wouldn’t expect quoted or unquoted configuration properties to work differently. As far as I can see, the OP was using quoted properties in the config file and tried to override them with an unquoted version.

If it’s not working with quotes, we have an issue, it should work.