quarkus: Problem with environment variables when using multiple datasources
Describe the bug
I belive there’s a problem with environment variables when using multiple datasources.
Looks like Quarkus isn’t recognizing the environment variables that have datasource namespaces, like quarkus.datasource."users".jdbc.url .
Or perhaps I’m not passing the variables correctly.
Expected behavior Quarkus should read the environment variables with the datasource namespace.
Actual behavior Quarkus isn’t reading reading the environment variables with the datasource namespace.
To Reproduce
- Build the project with
mvn clean package - Export the environment variables
- Execute the runnable jar.
- Quarkus says that the variables weren’t defined.
Configuration
quarkus.datasource."users".db-kind=postgresql
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.hibernate-orm."users".datasource=users
quarkus.hibernate-orm."users".packages=com.helesto.models.users
quarkus.hibernate-orm."users".database.generation = none
quarkus.hibernate-orm."users".sql-load-script=no-file
quarkus.hibernate-orm."users".log.sql=false
quarkus.datasource."orders".db-kind=mariadb
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}
quarkus.hibernate-orm."orders".datasource=orders
quarkus.hibernate-orm."orders".packages=com.helesto.models.orders
quarkus.hibernate-orm."orders".database.generation=none
quarkus.hibernate-orm."orders".sql-load-script=no-file
quarkus.hibernate-orm."orders".log.sql=false
Environment:
-
uname -a: Linux machine 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -
java -version: openjdk version “11.0.8” 2020-07-14 OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing) -
Quarkus version or git rev: 1.9.0.Final
-
mvnw --version: Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /home/helesto/.m2/wrapper/dists/apache-maven-3.6.3-bin/1iopthnavndlasol9gbrbg6bf2/apache-maven-3.6.3 Java version: 11.0.8, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64 Default locale: en_US, platform encoding: UTF-8 OS name: “linux”, version: “5.4.0-52-generic”, arch: “amd64”, family: “unix”
Test with my project - happy end
It’s possible to reproduce this problem with a project I created to test the use of multiple databases:
With this project, I did the following test:
- Build it with
./mvnw clean package - Start the docker containers of the databases (PostgreSQL and MariaDB):
docker run -d --name postgres-db -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=usersdb postgres
docker run -d --name mariadb-db -p 3306:3306 -e MYSQL_USER=maria -e MYSQL_ROOT_PASSWORD=maria -e MYSQL_PASSWORD=maria -e MYSQL_DATABASE=ordersdb mariadb
- Run the project in dev mode just to create the tables and insert some rows:
./mvnw compile quarkus:dev
- Enter inside the /target folder and run the runnable jar passing all parameters:
java -jar -Dquarkus.datasource.users.jdbc.url=jdbc:postgresql://localhost:5432/usersdb -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar
- Enter inside the http://localhost:8080/swagger-ui , execute the tests and confirm everything is ok.
Now the problem.
- I try to export just one variable with, like this one:
export QUARKUS_DATASOURCE_USERS_JDBC_URL=jdbc:postgresql://localhost:5432/usersdb
- Then I try to run the same command without this property (
quarkus.datasource."users".jdbc.url), just to test:
java -jar -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar
- Quarkus ignore my environment variable and I receive this error:
WARN [io.agr.pool] (main) Datasource 'users': Driver does not support the provided URL: prod-url
prod-urlis my default value:
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
Now my workaround
As Quarkus isn’t reading the property variables with the datasource namespace, I’ve created the variables below:
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}
For example:
- I export the POSTGRESQL_URL:
export POSTGRESQL_URL=jdbc:postgresql://localhost:5432/usersdb
- Now I run the application without passing this variable (
quarkus.datasource."users".jdbc.url) and everything works fine:
java -jar -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar
I’m using this solution to run my real application inside Kubernetes. That is, instead of passing QUARKUS_DATASOURCE_USERS_JDBC_URL and the other variables, I’m passing this variables I’ve created, like POSTGRESQL_URL.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (17 by maintainers)
Correct. We need the property defined in their dotted format, so we can disambiguate when the property is set in the environment variable name.
I’m having basically the same problem with namespaced property
quarkus.vault.credentials-provider."credentials-provider".database-credentials-rolein a Kubernetes environment using Vault database engine for dynamic credentials.My idea was to not include it in
application.propertiesat all but instead pass it only via KubernetesConfigMapasenvFromblock. I used the following in myConfigMap:…but I always end up with
It all works perfectly if I put
to
application.propertiesbut I would rather prefer not to have it there (keepapplication.propertiesas empty as possible).After some examination - and this issue - I come to the conclusion that it’s only possible to overwrite namespaced properties that already exist in
application.propertiesvia environment variables but not to introduce them.Can anyone confirm?
Great! Thanks!
In Quarkus we do canonicalize (to a degree) the quoting of property names, so
foo\.barshould be equivalent to"foo.bar", andfooshould be equivalent to"foo". If we’re seeing behavior that contradicts this, then I’d consider it a bug in the properties mapping.On the topic of environment variable names: it’s quite tricky because the mapping only goes one way. Your description is therefore slightly inaccurate: we do not convert
FOO_BARtofoo.barbut we do convertfoo.bartoFOO_BARwhen we probe the environment to see what properties are present. The conversion cannot be two-way because converting to environment vars is a lossy operation. For example we would considerfoo.bar_bazandfoo_bar.bazto be distinct configuration property names, but both of these would be mapped to an environment variable namedFOO_BAR_BAZ.Weird. As far as I know, we don’t treat quoted property names differently, in the sense that the quote is part of the name, meaning that you require the quote in the system property name to be able to override it. I’ve tried in one of my projects and I was only able to override it when using the quoted version.
The issue with environment variables is obvious, since you are not able to use quotes on their name it renders any override useful for quote property names. Also an environment variable name is just converted to its property name counter part, meaning that something like
FOO_BARgets converted tofoo.barto perform the lookup.@dmlloyd do you think we should convert quoted property names to their unquoted counterparts transparently? Right now, they are just treated as different properties.
Hum… can’t remember anything related, but I’m going to investigate.
@radcortez does it ring a bell?
I wouldn’t expect quoted or unquoted configuration properties to work differently. As far as I can see, the OP was using quoted properties in the config file and tried to override them with an unquoted version.
If it’s not working with quotes, we have an issue, it should work.