Get hashes from PyPI JSON API fails when index URL ends with slash

[michael-k]

When using https://pypi.org/simple/ instead of https://pypi.org/simple as index URL, pip-compile falls back to hashing files.

This is caused by passing the index URL to pip's PackageIndex and then calling package_index.pypi_url:

pip-tools/piptools/repositories/pypi.py

Lines 264 to 269 in 1e05d00

	package_indexes = (
	PackageIndex(url=index_url, file_storage_domain="")
	for index_url in self.finder.search_scope.index_urls
	)
	for package_index in package_indexes:
	url = f"{package_index.pypi_url}/{ireq.name}/json"

• pip itself only uses PackageIndex to initialize the constants PyPI and TestPyPI. I don't know what the recommended way is to convert between Simple API URLs and JSON API URLs.

• PEP 503 – Simple Repository API says that “PyPI’s base URL is https://pypi.org/simple/”. I don't see why the trailing slash would be considered wrong when used in an index URL.

• Sure, this shouldn't happen a lot for people who use PyPI.org. I've noticed it because AWS CodeArtifact generates/documents index URLs that end with /simple/ and then package_index.simple_url as used in Compute hashes from all index servers #1556 also doesn't work. But package_index.simple_url would have been my first approach for a potential fix of get sha256 hash from /simple (PEP503) endpoint #1135.

  Code Example

  from pip._internal.commands import create_command
  from pip._internal.models.index import PackageIndex
  
  command = create_command("install")
  options, _ = command.parse_args(["-i", "https://pypi.org/simple/"])
  session = command._build_session(options)
  finder = command._build_package_finder(options=options, session=session)
  
  for index_url in finder.search_scope.index_urls:
      package_index = PackageIndex(url=index_url, file_storage_domain="")
      print(package_index.simple_url)  # prints: https://pypi.org/simple/simple
      print(package_index.pypi_url)  # prints: https://pypi.org/simple/pypi

Environment Versions

1. OS Type: Linux
2. Python version: $ python -V: Python 3.10.6
3. pip version: $ pip --version: pip 22.2.2
4. pip-tools version: $ pip-compile --version: pip-compile, version 6.8.0

Steps to replicate

1. echo "click" > requirements.in
2. pip-compile --generate-hashes --verbose --index-url=https://pypi.org/simple/ requirements.in

Expected result

Log output

Using indexes:
  https://pypi.org/simple/

                          ROUND 1
Current constraints:
  click (from -r requirements.in (line 1))

Finding the best candidates:
  found candidate click==8.1.3 (constraint was <any>)

Finding secondary dependencies:
  click==8.1.3              requires -
------------------------------------------------------------
Result of round 1: stable, done

Generating hashes:
  click

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile --generate-hashes --index-url=https://pypi.org/simple/ requirements.in
#
click==8.1.3 \
    --hash=sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e \
    --hash=sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48
    # via -r requirements.in

Actual result

Log output

Using indexes:
  https://pypi.org/simple/

                          ROUND 1
Current constraints:
  click (from -r requirements.in (line 1))

Finding the best candidates:
  found candidate click==8.1.3 (constraint was <any>)

Finding secondary dependencies:
  click==8.1.3              requires -
------------------------------------------------------------
Result of round 1: stable, done

Generating hashes:
  click
    Couldn't get hashes from PyPI, fallback to hashing files
    Hashing click-8.1.3-py3-none-any.whl
      |████████████████████████████████| 100%
    Hashing click-8.1.3.tar.gz
      |████████████████████████████████| 100%

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile --generate-hashes --index-url=https://pypi.org/simple/ requirements.in
#
click==8.1.3 \
    --hash=sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e \
    --hash=sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48
    # via -r requirements.in

[atugushev]

Thanks for the issue @michael-k. Good catch!