rancher: Unable to update istio configuration in air-gapped environment in Rancher v2.5.x

What kind of request is this (question/bug/enhancement/feature request):

Bug

Steps to reproduce (least amount of steps as possible):

1. Provision a Rancher v2.5.5 instance and an air-gapped custom cluster (specifically it should not have access to github.com, I mimicked this in my non-air-gapped lab environment by adding the following to the coredns ConfigMap to prevent the resolution of github.com domains):

  github.com:53 {
    errors
    cache 30
    loop
    forward . 192.168.200.1
  }

1. Install Monitoring in the cluster: Cluster Explorer -> Apps & Marketplace -> Charts -> Monitoring
2. Install Istio in the cluster: Cluster Explorer -> Apps & Marketplace -> Charts -> Istio
3. Upgrade the Istio installation, changing the configuration, e.g. enable the Egress Gateway component.

Result:

Observe the upgrade has no effect and that the logs of the istioctl-installer Job Pod show the following error (in the case of mimicking air-gap with coredns ConfigMap change the exact lookup github.com error message will differ slightly):

Upgrade version check passed: 1.7.3 -> 1.7.3.

2021-01-13T14:30:17.398813Z     info    Error: failed to generate Istio configs from file [/app/istio-base.yaml] for the current version: 1.7.3, error: Get "https://github.com/istio/istio/releases/download/1.7.3/istio-1.7.3-osx.tar.gz": dial tcp: lookup github.com on 10.43.0.10:53: no such host

Error: failed to generate Istio configs from file [/app/istio-base.yaml] for the current version: 1.7.3, error: Get "https://github.com/istio/istio/releases/download/1.7.3/istio-1.7.3-osx.tar.gz": dial tcp: lookup github.com on 10.43.0.10:53: no such host

Other details that may be helpful:

There is an issue reporting this behaviour in the istio/istio GH repo at https://github.com/istio/istio/issues/28995

Behaviour originates from the istio upgrade command in the istioctl-installer Job https://github.com/rancher/istio-installer/blob/master/scripts/create_istio_system.sh

The error appears to come from here in the istio codebase, but at a first glance I could not determine the difference between the istioctl install and istioctl upgrade that means this is encountered only on the upgrade command, and not on the install.

Environment information

• Reproduced in Rancher v2.5.5, encountered by user in v2.5.2

gz#13932