galaxy: Sometimes a lot of HTTP status codes 520 when accessing Galaxy API

Bug Report

SUMMARY

I’m working on the Ansible changelog / porting guide build (ansible-community/antsibull#103). Both that build, and the ACD build itself, are querying the Galaxy API for all included collections (~60 of them). It often happens to me that I get a lot of 520 HTTP status codes (seems to be a Cloudflare internal error code):

WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/community/azure/versions/0.1.0/', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/community/azure/versions/0.1.0/', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/nxos/versions/?format=json&page=2', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/google/cloud/versions/0.10.1/', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/ios/versions/?format=json&format=json&format=json&page=4', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/junipernetworks/junos/versions/?format=json&format=json&format=json&format=json&page=5', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/ios/versions/?format=json&format=json&format=json&page=4', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/vyos/vyos/versions/?format=json&format=json&format=json&format=json&page=5', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/junipernetworks/junos/versions/?format=json&format=json&format=json&format=json&page=5', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/community/vmware/versions/?format=json&format=json&page=3', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/community/vmware/versions/?format=json&format=json&page=3', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/arista/eos/versions/?format=json&format=json&format=json&format=json&format=json&format=json&page=7', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/ios/versions/?format=json&format=json&format=json&page=4', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/nxos/versions/?format=json&format=json&format=json&page=4', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/iosxr/versions/?format=json&format=json&format=json&format=json&page=5', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/community/vmware/versions/?format=json&format=json&page=3', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/iosxr/versions/?format=json&format=json&format=json&format=json&page=5', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/cisco/nxos/versions/?format=json&format=json&format=json&format=json&page=5', params={'format': 'json'}) failed with status code 520, retrying...
WARNING: aio_session.get('https://galaxy.ansible.com/api/v2/collections/community/vmware/versions/?format=json&format=json&page=3', params={'format': 'json'}) failed with status code 520, retrying...

After adding code to retry the requests (with some increasing delay), it finally almost always completes (before I had to run it 2-10 times until it completed).