test-infra: Test failures starting 22 September due to network timeouts

Google has confirmed the issue in the support case: After reviewing the information you provided, we believe that you may be affected by a known issue: We have identified a Networking connecting issue impacting the GKE Docker workload. This is a high priority issue that we're working to resolve as soon as possible. Some customers may be experiencing a connection failure in Docker workflow to Fastly destinations and may receive a timeout error.

I would like for us to roll this change back, but I’m wary that maybe we’ll need it again someday.

So as to avoid a protracted roll out cycle, I’m planning on doing the following:

• gate the workaround behind an env var BOOTSTRAP_MTU_WORKAROUND, defaulted in-image to true… this will let us roll out changes selectively or globally by job config changes (1 PR) instead of propagating to kubekins (bootstrap change PR + bootstramp bump PR + kubekins bump PR)
• selectively disable the workaround on jobs that were broken before the workaround was in place via job config changes (add env stanzas setting BOOTSTRAP_MTU_WORKAROUND=false
• disable the workaround for all jobs by setting it in the preset-dind-enabled preset
• default the workaround to false in-image
• …leaving us the option to enable the workaround by default using the same “env var in preset” approach if this problem surfaces again

I’m told there is a fix scheduled for Thursday.

Current suggested workaround is add an initContainer to the Docker in Docker workload like this:

initContainers:                                                                                                                                                                                         
  - name: workaround                                                                                                                                                                                      
    image: k8s.gcr.io/build-image/debian-iptables-amd64:buster-v1.6.7                                                                                                                                     
    command:                                                                                                                                                                                              
    - sh                                                                                                                                                                                                  
    - -c                                                                                                                                                                                                  
    - "iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu"                                                                                                    
    securityContext:                                                                                                                                                                                      
      capabilities:                                                                                                                                                                                       
        add:                                                                                                                                                                                              
        - NET_ADMIN                                                                                                                                                                                       
      privileged: true

@alvaroaleman that was my question too, we’re on GKE. We observed it in one zone on Friday (europe-west4-b), and excluded that zone from our egress.

Over the weekend it started in west4-a and west4-c as well so certainly feels like a change was being rolled out per-zone.

We have an open ticket with Google but haven’t had any confirmation yet… (not convinced we will get any either to be honest). Like you, whilst we have a work around - I’d really like to understand why now.

https://status.cloud.google.com/incidents/QSirAFiyN5yMeeE6GNxq is listed as resolved

@alvaroaleman reports that setting TCP MSS clamping worksaround the issue, i.e. to execute once in the pod:

That is an educated but unverified guess based on prior failures with similar symptoms, fwiw.