pip-audit: Ignore duplicate requirements

Bug description

With using pip-tools workflow for layered requirements pip-audit fails because of duplicate requirements.

Reproduction steps

echo "django" > a.in
pip-compile -q --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras a.in
echo -e "-c a.txt\ndjango-debug-toolbar" > b.in
pip-compile -q --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras b.in
pip-audit -r a.txt -r b.txt --require-hashes

Expected behavior

No errors

Screenshots and logs

DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='a.txt' mode='r' encoding='UTF-8'>, <_io.TextIOWrapper name='b.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=1, fix=False, require_hashes=True, index_url='https://pypi.org/simple/', extra_index_urls=[], skip_editable=False, no_deps=False, output=PosixPath('stdout'), ignore_vulns=[])
DEBUG:pip_audit._cli:Auditing asgiref (3.6.0)
DEBUG:pip_audit._cli:Auditing django (4.1.7)
DEBUG:pip_audit._cli:Auditing sqlparse (0.4.3)
ERROR:pip_audit._cli:package asgiref has duplicate requirements: asgiref==3.6.0 (from RequirementLine(line_number=7, line='asgiref==3.6.0     --hash=sha256:71e68008da809b957b7ee4b43dbccff33d1b23519fb8344e33f049897077afac     --hash=sha256:9567dfe7bd8d3c8c892227827c41cce860b368104c3431da67a0c5a65a949506', filename=PosixPath('/tmp/tmpifyxbix9')))

Platform information

OS name and version: Arch Linux
pip-audit version (pip-audit -V): pip-audit 2.5.2
Python version (python -V or python3 -V): Python 3.10.10
pip version (pip -V or pip3 -V): pip 23.0.1

Additional context

About this issue

Original URL
State: open
Created a year ago
Comments: 17 (10 by maintainers)

Most upvoted comments

Much appreciated, thank you!

From a quick look, this stems from the current “fast path” behavior we have for --require-hashes. I believe it’ll get resolved via #540; cc @tetsuo-cpp for opinions.

woodruffw on Mar 21, 2023

Gotcha, I understand now: I forgot that we left this open only for the --fix part, and that we resolved the duplicate requirement issue on audits. Sorry for the regression here; I’ll look into a fix.

woodruffw on Jul 10, 2023

@woodruffw --fix complaining about duplicates isn’t new to 2.5.0 so this is probably not as urgent.

tetsuo-cpp on Mar 24, 2023

@tetsuo-cpp

echo "pyjwt==1.7.1" > requirements.in
pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras
cat requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras
#
pyjwt==1.7.1 \
    --hash=sha256:5c6eca3c2940464d106b99ba83b00c6add741c9becaec087fb7ccdefea71350e \
    --hash=sha256:8d59a976fb773f3e6a39c85636357c4f0e242707394cadadd9814f5cbaa20e96
    # via -r requirements.in

pip-audit --fix --no-deps -r requirements.txt
cat requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes --resolver=backtracking --strip-extras
#
pyjwt==1.7.1 \
    --hash=sha256:5c6eca3c2940464d106b99ba83b00c6add741c9becaec087fb7ccdefea71350e \
    --hash=sha256:8d59a976fb773f3e6a39c85636357c4f0e242707394cadadd9814f5cbaa20e96
# via -r requirements.in
    # pip-audit: subdependency explicitly fixed
pyjwt==2.4.0

It also fails without --no-deps, just --fix

q0w on Mar 24, 2023