kops: Route53 ARN has incorrect partition in govcloud

1. What kops version are you running? The command kops version, will display this information. Version 1.15.2 (git-ad595825a)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag. Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.3”, GitCommit:“06ad960bfd03b39c8310aaf92d1e7c12ce618213”, GitTreeState:“clean”, BuildDate:“2020-02-11T18:14:22Z”, GoVersion:“go1.13.6”, Compiler:“gc”, Platform:“linux/amd64”}

3. What cloud provider are you using? AWS GovCloud

4. What commands did you run? What is the simplest way to reproduce this issue? Creating a cluster in an existing VPC with existing subnets and an existing Route53 Zone. The zone was specified as a Zone ID.

5. What happened after the commands executed?

W0221 23:03:15.427827   12438 executor.go:130] error running task "IAMRolePolicy/<redacted>" (9m59s remaining to succeed): error creating/updating IAMRolePolicy: MalformedPolicyDocument: Partition "aws" is not valid for resource "arn:aws:route53:::hostedzone/<redacted>".
        status code: 400, request id: <redacted>
W0221 23:03:15.427860   12438 executor.go:130] error running task "DNSName/<redacted>" (9m59s remaining to succeed): error creating ResourceRecordSets: NoSuchHostedZone: The specified hosted zone does not exist.

6. What did you expect to happen? The route53 arn should use the partition associated with the region, aws-us-gov.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2020-02-21T22:55:11Z"
  generation: 3
  name: <redacted>
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudLabels:
    Owner: Ops
    Team: Ops
  cloudProvider: aws
  configBase: s3://<redacted>/<redacted>
  dnsZone: <redacted>.
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-us-gov-west-1b-1
      name: "1"
    - instanceGroup: master-us-gov-west-1b-2
      name: "2"
    - instanceGroup: master-us-gov-west-1b-3
      name: "3"
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-us-gov-west-1b-1
      name: "1"
    - instanceGroup: master-us-gov-west-1b-2
      name: "2"
    - instanceGroup: master-us-gov-west-1b-3
      name: "3"
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - <redacted>
  kubernetesVersion: 1.15.9
  masterInternalName: api.internal.<redacted>
  masterPublicName: api.<redacted>
  networkCIDR: <redacted>
  networkID: vpc-<redacted>
  networking:
    weave:
      mtu: 8912
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - <redacted>
  subnets:
  - cidr: <redacted>
    id: subnet-<redacted>
    name: us-gov-west-1a
    type: Private
    zone: us-gov-west-1a
  - cidr: <redacted>
    id: subnet-<redacted>
    name: us-gov-west-1b
    type: Private
    zone: us-gov-west-1b
  - cidr: <redacted>
    id: subnet-<redacted>
    name: utility-us-gov-west-1a
    type: Utility
    zone: us-gov-west-1a
  - cidr: <redacted>
    id: subnet-<redacted>
    name: utility-us-gov-west-1b
    type: Utility
    zone: us-gov-west-1b
  topology:
    dns:
      type: Private
    masters: private
    nodes: private

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-02-21T22:55:11Z"
  labels:
    kops.k8s.io/cluster: <redacted>
  name: master-us-gov-west-1b-1
spec:
  additionalSecurityGroups:
  - sg-<redacted>
  image: ami-<redacted>
  machineType: t3.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-gov-west-1b-1
  role: Master
  subnets:
  - us-gov-west-1b
  tenancy: dedicated

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-02-21T22:55:11Z"
  labels:
    kops.k8s.io/cluster: <redacted>
  name: master-us-gov-west-1b-2
spec:
  additionalSecurityGroups:
  - sg-<redacted>
  image: ami-<redacted>
  machineType: t3.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-gov-west-1b-2
  role: Master
  subnets:
  - us-gov-west-1b
  tenancy: dedicated

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-02-21T22:55:11Z"
  labels:
    kops.k8s.io/cluster: <redacted>
  name: master-us-gov-west-1b-3
spec:
  additionalSecurityGroups:
  - sg-<redacted>
  image: ami-<redacted>
  machineType: t3.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-gov-west-1b-3
  role: Master
  subnets:
  - us-gov-west-1b
  tenancy: dedicated

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-02-21T22:55:11Z"
  labels:
    kops.k8s.io/cluster: <redacted>
  name: nodes
spec:
  additionalSecurityGroups:
  - sg-<redacted>
  image: ami-<redacted>
  machineType: t3.2xlarge
  maxSize: 3
  minSize: 3
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - us-gov-west-1a
  - us-gov-west-1b
  tenancy: dedicated

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here. Relevant output:

I0221 23:11:41.338498   12819 iamrolepolicy.go:147] Creating IAMRolePolicy
I0221 23:11:41.338515   12819 iamrolepolicy.go:175] PutRolePolicy RoleName=masters.<redacted> PolicyName=masters.<redacted>: {
...
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets",
        "route53:GetHostedZone"
      ],
      "Resource": [
        "arn:aws:route53:::hostedzone/<redacted>"
      ]
    },
...

Other ARNs in the same output, such as for S3, properly include the aws-us-gov partition.

9. Anything else do we need to know?