platformio-core: `lib_extra_dirs = ...` no longer works with 5.3.0a3

PlatformIO Core. If you’ve found a bug, please provide an information below.

Configuration

Operating system: Fedora 35 (Container Image)

PlatformIO Version (platformio --version): PlatformIO Core, version 5.3.0a3

Description of problem

When lib_extra_dirs directory already has a library installed, pio run will still do an installation into the .pio/libdeps/${env}/

(plus, in case of non-registry dependency, it would also mean it will fetch the git repo twice)

This was not the case when using the 5.2.x Probably related to the #4198?

Steps to Reproduce

Create a dummy project via pio project init -b esp32dev
Go into the project and manually install some lib

$ mkdir my_libraries
$ pio lib -d my_libraries/ install 'bblanchon/ArduinoJson@6.19.3'
Library Storage: /home/builder/esp32dev/my_libraries
Library Manager: Installing bblanchon/ArduinoJson @ 6.19.3
Unpacking  [####################################]  100%
Library Manager: ArduinoJson @ 6.19.3 has been installed!

Add both dependency and the lib_extra_dirs entry to the environment

lib_deps =
    bblanchon/ArduinoJson@6.19.3
lib_extra_dirs =
    my_libraries/

pio run

Processing esp32dev (platform: espressif32; board: esp32dev; framework: arduino)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Library Manager: Installing bblanchon/ArduinoJson @ 6.19.3
^C
Aborted!
^C⏎

Expected Results

Don’t install things twice and re-use the existing library storage.

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
lib_deps =
    bblanchon/ArduinoJson@6.19.3
lib_extra_dirs =
    my_libraries/

Source file to reproduce issue:

#include <Arduino.h>

void setup() {
}

void loop() {
}

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 18 (16 by maintainers)

Most upvoted comments

Will lib_extra_dirs still be there in future

We never remove any functionality that can break existing projects. This is our metrology that was introduced in 2014. Any project created in 2014 is fully compatible with the latest PlatformIO Core. We don’t plan to change it.

This issue was related to the Package Manager (pio pkg) command and lib_deps. We will encourage developers to use a declarative approach for critical safety embedded development based on lib_deps and Semantic versioning. See upcoming pio pkg command ( https://docs.platformio.org/en/latest/core/history.html ) and specifications and version requirements.

This approach does not depend on lib_extra_dirs and PlatformIO will never manage automatically (check updates, etc) custom user storage until they use pio pkg --globa --storage-dir /path/to/extra-storage.

We would be thankful if you switch Tasmota to the latest development version of PlatormIO Core and report any issues. So, we will be confident that the upcoming stable release will not break popular projects. We even added them to our docs ( https://docs.platformio.org/en/latest/tutorials/index.html#projects ).

ivankravets on Apr 3, 2022

I’ll need to play around with it some more, as I was going to ask about search and turns out it is already there 😃 Thanks for the continued support!

mcspr on Mar 31, 2022

Ok. $PROJECT_DIR/lib worked with the proposed script above, I’ll update the project (unless lib/ is going to be removed as well at some point 😃

Only cleanup I found required is this additional step that works like original ‘multiple’ config value of lib_deps

- lib_deps = config.get("common", "lib_deps")
+ lib_deps = config.parse_multi_values(config.get("common", "lib_deps"))

mcspr on Mar 31, 2022

Sorry, PlatformIO is already complex in terms of managing libraries. That was our mistake with adding support for “extra dirs” and global libraries. Adding “yet another” storage logic for library dependencies will finally destroy all our efforts on improving the developer experience. The goal is to reduce this complexity and encourage people to use SemVer declarative approach. We’ve not yet found a better methodology to manage dependencies.

I close this issue. If you don’t want to use default storage and declarative approach with lib_deps and SemVer, please check this comment ( https://github.com/platformio/platformio-core/issues/4206#issuecomment-1078881377 ). This example allows you to do everything that you want with storage and packages.

Thanks for understading.

ivankravets on Mar 31, 2022