pants: Tailor Command not detecting new python_sources with top-level dependencies files

After doing some tests, I went with the alternative to add a sed command in my build script that will add a # to ignore one line per BUILD file at the top of each microservices.

So my BUILD file will look like this in the repo, so all tailor commands work perfectly and all tests load all the files like json, ini and anything that could be in any project.

poetry_requirements(
    name="poetry",
)

python_sources()

files(
    name="files",
    sources=[
        "*/**/*",
        "poetry.lock",
        "pyproject.toml",
        "!*/**/*.py",
    ],
)

docker_image(
    name="docker",
    dependencies=[":files"]
)

and the sed command in my build script replace the line "!*/**/*.py", for # "!*/**/*.py", which import all python file before doing the publish command. So far so good! If you have a better idea, please tell me. This is a workaround and if there is another possibility using Pants to do the most hermetic approach between testing and packaging, that would be sweet.

Just to give you more background on us, we have 20+ microservices with libraries that we migrated to a mono repo pattern. We started using pants to bring standardization on the Linting, Unit Testing and a generic solution to Package and Publish our Containers. I tried the PEX way first and realize that some of our projects are old or not implemented in a “standards” way which was resulting in a lot of conflicts, missing imports, missing files and broken entry points (since some projects had up to 4 entry points for stuff like migrations, management, etc) So I changed our approach by keeping everything like Formatting, Linting and Testing in a pure Pants approach but the Packaging and Publishing process to a kind of Hybrid style (load all files and Build all Dockerfile the way we were doing before but asking pants to do it) that’s why I need all the files and python files loaded up ONLY on the packaging.

I know this is not the “pants way” but this is the solution we found that was working the best currently. Right now, all the tests are working great and the full pipeline is using Toolchain to remote cache, the publishing is working great and nothing broke on the end.

The issue is bugging us at the moment when developers are creating new packages, the BUILD files are not generated along the new package and cause conflicts since the Unit tests are not picking the package without BUILD files at their root. Currently, they add manually the BUILD files every time but we would like to use the tailor command to save us time.

---

If there is a possibility to ignore a files or dependencies depending on the runtime since we only need the files dependencies when we package and publish everything,

Pardon my confusion, I don’t follow here. What would you like to achieve?

Sorry for the long text but I think this will help the issue to have more context on our current state.

To answer your question: When I remove completely the files and the docker_image from my BUILD file, the tailor and everything else continue to work correctly. Since I only need those on the package goal if there is a possibility to ignore them for everything else and only add on the package command (using something like environment or tags) this would be great.

---

For this:

docker_image(
    name="docker",
    # give me everything found in the directory, recursively picking up files and sources
    dependencies=["//src/services/project::"]  
)

if there is no need to have the files dependencies target and this will make pants to see all the new Python files unowned correctly so tailor can create the BUILD file, yes this would be perfect and this would clear all my project directory BUILD files clean.

Just tested on the pants-issue Repo with the following

poetry_requirements(
    name="poetry",
)

python_sources(name="package_sources", sources=["*/**/*.py"])

files(
    name="files",
    sources=[
        "*/**/*",
        "poetry.lock",
        "pyproject.toml",
        "!*/**/*.py",
    ],
)

docker_image(
    name="docker",
    dependencies=[":files", ":package_sources"]
)

and tailor only added a new python_sources() in src/services/project/BUILD and nothing in src/services/project/package/not_working

Am I missing something ? (running ./pants tailor ::)

Sorry for the late reply, I will test that ASAP and update the issue if this solve our needs