pydantic: MissingPath / Potential{File,Directory}Path

Feature Request

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

$ python3 -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.4a1
            pydantic compiled: False
                 install path: /home/[user]/git/pydantic/pydantic
               python version: 3.7.5 (default, Nov 20 2019, 09:21:52)  [GCC 9.2.1 20191008]
                     platform: Linux-5.3.0-29-generic-x86_64-with-Ubuntu-19.10-eoan
     optional deps. installed: []

Related to: #10

A better way to validate if a path does not exists. This could then also be used without much effort to check if a path either is a file/dir or could be created (meaning it is not a directory/char device/…)

#! /usr/bin/env python3
from pathlib import Path
from typing import Union
from pydantic import BaseModel, FilePath, DirectoryPath
from pydantic.validators import path_validator


class MissingPath(Path):
    @classmethod
    def __get_validators__(cls) -> 'CallableGenerator':
        yield path_validator
        yield cls.validate

    @classmethod
    def validate(cls, value: Path) -> Path:
        if value.exists():
            raise ValueError("path exists")

        return value


PotentialFilePath = Union[FilePath, MissingPath]
PotentialDirectoryPath = Union[DirectoryPath, MissingPath]


class FileWriter(BaseModel):
    file_path: PotentialFilePath  # alternatively Path+custom validator works, but gets bulky

    def write(self, text):
        with self.file_path.open("w") as f:
            f.write(text)

fw = FileWriter(file_path=Path("test.txt"))
fw.write("hello")

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 15 (5 by maintainers)

Most upvoted comments

From what I can tell, this is fixed on main and will be included in v2 now, but is called pydantic.types.NewPath. I think the equivalent of PotentialFilePath = Union[FilePath, MissingPath], for example, would be Union[pydantic.types.FilePath, pydantic.types.NewPath]. You can try this now in the alpha pre-release of v2.

If I’ve misinterpreted the issue or the above is otherwise not right, please comment here and we can re-open the issue if appropriate.

Hi - any progress on this feature request? This would really help my use case.

@NovaNekmit Please take a look if this is matches your expectations: https://github.com/xkortex/pydantic/tree/feature/missing-path-1254

Actually, the above logic isn’t totally correct, have to resolve symlinks and whatnot. Also fun fact, did you know os.path.abspath() and Path.absolute() have different behavior? That’s fun. Also, this is also fun: https://bugs.python.org/issue25012.

Yeah, I’m going to need to use some more sophisticated test fixtures at some point.

def is_potential_path(path: Path, exist_ok=False):
    path = Path(path).expanduser().absolute()
    if path.exists():
        if not exist_ok:
#             print(f"<!> file exists: {path}")
            return False
        else:
            return True
    ancestor = nondir_ancestor(path.parent)
    if ancestor is not None:
#         print(f"<!> ancestor is not a dir: {ancestor}" )
        return False
    return True

def nondir_ancestor(path: Path) -> Optional[Path]:
    """Find an ancestor which isn't a dir, and return it so we can report that path to the error. 
    Otherwise return None"""
    path = path.expanduser().resolve()
    if path.is_dir():
        return None
    if path.exists():
        return path
    return nondir_ancestor(path.parent)
       ~/does/not/exist: True
           /sure/we/can: True
                    yup: True
             ./also/yes: True
              ../indeed: True
               ~/.zshrc: False
                      /: False
          /dev/tty/nope: False
/dev/tty/../actually_ok: True

If we set exist_ok=True, then the PotentialPath can be an extant file/dir, but it must have a valid ancestor:

       ~/does/not/exist: True
           /sure/we/can: True
                    yup: True
             ./also/yes: True
              ../indeed: True
               ~/.zshrc: True
                      /: True
          /dev/tty/nope: False
/dev/tty/../actually_ok: True

My rationale behind PotentialFilePath was that it either already exists as file or could exist if it were to be created (optionally including making parent paths) without any operation running into a case where the name is “blocked” by a “wrong” type.

Primary usecase being files that would be created if they do not exist yet.

Example of what I was thinking for ./foo/bar.json:

Allowed:

  • ./foo/bar.json exists (FilePath)
  • ./foo exists, but does not contain anything named bar.json (MissingPath)
  • ./foo does not exist (MissingPath)

Disallowed:

  • ./foo/bar.json is a directory or device
  • ./foo is a file or device, so no directory could be created there (edgecase, may be out of scope to detect this)

I think no fancy, especially as I would assume StrictPath would just require an argument that is a path object, not string etc.

I think let’s start with MissingPath and PotentialPath (does not exist but parent exists).

While I’m in this space, is there any interest in something like a StrictPath, which restricts certain characters to facility cross-platform filenames (e.g. no ? < > : * | ” ), or GlobPath/FnMatchPath/ReMatchPath, which expect the path to match some pattern?

Something like

class ImageSet(BaseModel): 
    images: List[FnMatchPath["*.jpg"]

Useful? Too Fancy?

I’m playing around with this and realizing, wouldn’t PotentialFilePath just be a Path (pathlib.Path) at that point? Posix paths are kinda weird in that any character sans / and \0 are valid (and actually PosixPath('\0') works), so path.exists() || path.not_exists() is tautology, which means you just need path_validator, which I think the annotation foo: Path uses.

Windows may behave differently but I have to get an environment set up before I can play around with that.

I’m still interested in such a feature. I might be able to PR if I can spare some time. Have to take a gander and see how they do the test harness for the existing Path fields. Should be a pretty small PR.