warehouse: Packages randomly missing from simple index

Describe the bug We upload the AWS Python packages (awscli, boto3, and botocore) each weekday with changes to keep the tools up-to-date with the company’s services. We realize that generates a significant number of entries in the PyPI index, so we’re an outlier to most projects.

Recently, we’ve started noticing that periodically (every 2-3 weeks) one or more of the uploads will being missing from either the /project page or the /simple index, but never both. We’re confident the uploads are succeeding from our twine logs and the fact it’s always upload in at least one of those two pages. /project isn’t a major issue, but when the /simple entry is missing, users are receiving errors in pip preventing them from installing the day’s updates.

I don’t have a strong understanding of the sync mechanism, but it seems whatever job is populating these in PyPI may be dying or halting. Whenever these issues arise, we’ve been waiting for the next day’s release which seems to fix all entries immediately. The goal of this ticket is to get some more insight into what’s happening and if there are actions we can take to mitigate it.

Expected behavior Packages to be available in the /simple/<project> index within a reasonable amount of time after upload.

To Reproduce https://pypi.org/simple/boto3/ is currently missing entries for 1.20.9. This will likely be remedied with our next release at ~1200 PT but is currently causing issues for users installing with pip.

My Platform Our build and uploads are using:

setuptools==57.5.0
wheel==0.37.0
twine==1.15.0