magento2: Functional api test unable to load install-config-mysql database credentials

OK, ive had a look at the API Functional test code and how it works and it exposes a fundamental issue with the way the API Functional tests work.

Lets consider the integration tests first. When they are run, they use the install-mysql-config.php and the test framework installs a bastardised runtime of magento into dev/tests/integration/tmp/sandbox---------- and creates a separate database as defined in install-mysql-config.php. The tests are all run in the context of this sandbox test environment so as to not affect any data in the main development database. This works fine and is well understood.

Now, the API Functional tests. Well, these seem to follow the same approach, as there is a install-mysql-config.php file detailing a separate database however the bootstrap functions in the api functional tests operate differently to the integration tests. The test framework seems to have been implemented by a mixture of people who don’t understand the difference between api functional and integration tests, and people that have hacked it to appear to work. Let me explain.

So, the difference between the two test types : whereas the integration tests run in their own sandbox, you cannot do this for the api functional tests so easily. The API Functional tests are essentially very simple REST/SOAP/GRAPHQL clients which makes a call to a Magento HTTP(s) endpoint. Had the test framework been configured to work in the same way as the integration tests with its own sandbox using details in install-mysql-config.php, then it would be ONLY the REST/SOAP/GRAPHQL clients that run in this sandbox. You would have a sandboxed database in existence for the test client, which then makes a call to a HTTP(s) endpoint somewhere else. but where??? There is only one HTTP(s) based Magento endpoint and that is the one configured in app/etc/env.php. We do not have a separate webserver setup that could serve the sandboxed version of the api-functional test installation.

I believe at this point the Magento team may well have just thrown their hands in the air and decided that the API functional tests should not run in a sandbox environment, and adjusted the test framework bootstrap code to operate on the main database. They did leave in a couple of nasty features though

• if the <const name="TESTS_CLEANUP" value="disabled"/> setting is enabled in phpunit.xml then the system attempts to uninstall the system defined in app/etc/env.php
• if the <const name="TESTS_MAGENTO_INSTALLATION" value="enabled"/> setting is enabled in phpunit.xml then the system reinstalls into the main database defined in app/etc/env.php

So, what do we do. The current API functional tests intentionally ignore the install-mysql-config.php data and use the app/etc/env.php details instead. The tests operate on the HTTP(s) endpoint configured in app/etc/env.php with the result being the tests are performed on the main development database and not a sandboxed version. People assume it uses a sandbox database but it doesn’t, it uses whichever database is configured behind the HTTP(s) endpoint. Their development database will be being polluted with data from the API Functional tests, if not cleaned up properly by the test.

It is fairly straight forward to adjust the test framework to use the install-mysql-config.php settings and generate a sandbox for the REST/SOAP/GRAPHQL client to run in, but we would also need to either adjust the main app/etc/env.php to temporarily use the sandbox details from install-mysql-config.php for the duration of the test, or create a separate web server config that points at the sandbox installation.

you could have two env.php files and use a symbolic link to switch between them. It could be triggered automatically by a test runner script. Lets see if Magento core comes back with any comments. I agree its not ideal.

Hi, @ihor-sviziev I wondered if you were aware of the issues with the API-Functional test framework as detailed in my post above. Can you copy anyone in, in the test team that may need to be aware of this. Its been in this broken/working only by chance state for a while now, and I am sure because of the bug, regularly wipes peoples development databases when they first try to run the intergration tests, and pollutes their development database with test data as the test framework uses the wrong database.

I raised https://github.com/magento/magento2/issues/27213 on it a while ago but got no traction.

A little bit more research that confirms the fundamental broken’ness of the API Functional tests.

If the following settings are set in phpunit.xml

<const name="TESTS_CLEANUP" value="disabled"/>
<const name="TESTS_MAGENTO_INSTALLATION" value="disabled"/>

Then running the API-Functional tests will run them against the Magento HTTP(s) endpoint as defined in the setting <const name="TESTS_BASE_URL" value="http://localhost:8080/"/>. It will not attempt to uninstall or reinstall the database.

if either of the TESTS_CLEANUP (ununstalls magento) or TESTS_MAGENTO_INSTALLATION (installs magento) are set to enabled, then because of this bug, then the database defined in app/etc/env.php is DROPPED and then the database defined in install-config-mysql.php is installed. This is probably not what you want and will most likely mess up your development node database.

So, two ways to run API Functional tests…

Run the API Functional Tests against Magento configured to use your development database.

• Leave your app/etc/env.php alone.
• No need to set anything in install-config-mysql.php. It is not used in this configuration.
• Set TESTS_CLEANUP to disabled in phpunit.xml
• Set TESTS_MAGENTO_INSTALLATION to disabled in phpunit.xml
• Set TESTS_BASE_URL, TESTS_WEBSERVICE_USER, TESTS_WEBSERVICE_APIKEY in phpunit.xml to match the credentials on your development node. TESTS_BASE_URL is the web address of your dev node installation.
• Run the API-Functional test. The tests will be run against your development node.
• Be aware it may pollute your dev database with test data.

Run the API Functional Tests against Magento configured to use a sandbox database.

• Prior to running the test. Update your app/etc/env.php to point to your sandbox database.
• Set TESTS_CLEANUP to enabled in phpunit.xml
• Set TESTS_MAGENTO_INSTALLATION to enabled in phpunit.xml
• Set TESTS_BASE_URL, TESTS_WEBSERVICE_USER, TESTS_WEBSERVICE_APIKEY in phpunit.xml to match the credentials as set in install-config-mysql.php. TESTS_BASE_URL is the web address of your dev node installation.
• Run the API-Functional test.
• It will uninstall the database defined in app/etc/env.php (Which should point to your sandbox right!).
• It will reinstall the database defined in install-config-mysql.php (the items in this file are directly transposed into command line arguments in the setup:install command so adjust to suit, but this should be the command to install magento into your sandbox database).
• Because you updated app/etc/env.php (your development node), the Magento HTTP(s) endpoint that is used in your test will act on your sandbox database.
• At the end of the test, revert the app/etc/env.php back to your original developer settings.

I’ll take a look at this later today as I almost had a fix working when I last looked at this.