k8s-bigip-ctlr: Ingress controller can not update http_to_https redirect

Description

After adding two or more ingress resources to the cluster, the ingress controller is unable to apply its config.

Kubernetes Version

Kubernetes cluster is running in version 1.17.2.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:22:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Controller Version

Controller is running in version v1.13.0.

$ kubectl logs -f k8s-bigip-ctlr-7b5cf67f76-k8m27
2020/05/08 10:53:43 [INFO] Starting: Version: v1.13.0, BuildInfo: n2182-642304053
2020/05/08 10:53:43 [INFO] ConfigWriter started: 0xc00030c1e0
2020/05/08 10:53:43 [INFO] Started config driver sub-process at pid: 13
2020/05/08 10:53:43 [ERROR] EOF
2020/05/08 10:53:43 [ERROR] [AS3] Error in validating declaration
2020/05/08 10:53:43 [INFO] NodePoller (0xc0001e21b0) registering new listener: 0x11c1b50
2020/05/08 10:53:43 [INFO] NodePoller started: (0xc0001e21b0)

BIG-IP Version

BIG-IP device is running on version BIG-IP 15.1.0 Build 0.0.31 Final

Diagnostic Information

Controller logs

2020/05/08 10:54:44 [INFO] Wrote 2 Virtual Server and 0 IApp configs
2020/05/08 10:54:44 [WARNING] Overwriting existing entry for backend {ServiceName:ingress1-kb-http ServicePort:<PORT> Namespace:elasticsearch}
2020/05/08 10:54:44 [INFO] Wrote 2 Virtual Server and 0 IApp configs
2020/05/08 10:54:44 [INFO] [2020-05-08 10:54:44,797 f5_cccl.resource.resource INFO] Creating ApiInternalDataGroup: /<PARTITION>/https_redirect_dg
2020/05/08 10:54:44 [WARNING] Overwriting existing entry for backend {ServiceName:service1-kb-http ServicePort:<PORT> Namespace:<NAMESPACE>}
2020/05/08 10:54:44 [WARNING] Overwriting existing entry for backend {ServiceName:service1-kb-http ServicePort:<PORT> Namespace:<NAMESPACE>}
2020/05/08 10:54:44 [INFO] Wrote 2 Virtual Server and 0 IApp configs
2020/05/08 10:54:44 [INFO] [2020-05-08 10:54:44,905 f5_cccl.resource.resource INFO] Creating ApiIRule: /<PARTITION>/http_redirect_irule_443
2020/05/08 10:54:44 [INFO] Wrote 2 Virtual Server and 0 IApp configs
2020/05/08 10:54:46 [WARNING] Overwriting existing entry for backend {ServiceName:service2-service ServicePort:9000 Namespace:<NAMESPACE>}
2020/05/08 10:54:46 [WARNING] Overwriting existing entry for backend {ServiceName:service2-service ServicePort:9000 Namespace:<NAMESPACE>}
2020/05/08 10:54:46 [INFO] Wrote 2 Virtual Server and 0 IApp configs
2020/05/08 10:54:48 [INFO] Wrote 2 Virtual Server and 0 IApp configs
2020/05/08 10:54:52 [INFO] [2020-05-08 10:54:52,408 f5_cccl.resource.resource INFO] Updating ApiInternalDataGroup: /<PARTITION>/https_redirect_dg
2020/05/08 10:54:52 [ERROR] [2020-05-08 10:54:52,570 f5_cccl.resource.resource ERROR] HTTP error(400): CCCL resource(ApiInternalDataGroup) /<PARTITION>/https_redirect_dg.
2020/05/08 10:54:52 [ERROR] [2020-05-08 10:54:52,570 f5_cccl.service.manager ERROR] F5CcclResourceRequestError - 400 Unexpected Error: Bad Request for uri: https://F5_IP_ADDRESS:443/mgmt/tm/ltm/data-group/internal/~<PARTITION>~https_redirect_dg/
2020/05/08 10:54:52 [INFO] Text: u'{"code":400,"message":"0107074b:3: Unable to change data group (/<PARTITION>/https_redirect_dg) type.  Must remove existing entries first.","errorStack":[],"apiError":3}'
2020/05/08 10:54:52 [ERROR] [2020-05-08 10:54:52,571 f5_cccl.service.manager ERROR] Resource /<PARTITION>/https_redirect_dg update error, requeuing task...
2020/05/08 10:54:52 [ERROR] [2020-05-08 10:54:52,833 __main__ ERROR] Error applying config, will try again in 1 seconds
2020/05/08 10:54:53 [INFO] [2020-05-08 10:54:53,132 f5_cccl.resource.resource INFO] Updating ApiInternalDataGroup: /<PARTITION>/https_redirect_dg
2020/05/08 10:54:53 [ERROR] [2020-05-08 10:54:53,390 f5_cccl.resource.resource ERROR] HTTP error(400): CCCL resource(ApiInternalDataGroup) /<PARTITION>/https_redirect_dg.
2020/05/08 10:54:53 [ERROR] [2020-05-08 10:54:53,390 f5_cccl.service.manager ERROR] F5CcclResourceRequestError - 400 Unexpected Error: Bad Request for uri: https://F5_IP_ADDRESS:443/mgmt/tm/ltm/data-group/internal/~<PARTITION>~https_redirect_dg/
2020/05/08 10:54:53 [INFO] Text: u'{"code":400,"message":"0107074b:3: Unable to change data group (/<PARTITION>/https_redirect_dg) type.  Must remove existing entries first.","errorStack":[],"apiError":3}'
2020/05/08 10:54:53 [ERROR] [2020-05-08 10:54:53,390 f5_cccl.service.manager ERROR] Resource /<PARTITION>/https_redirect_dg update error, requeuing task...
2020/05/08 10:54:54 [INFO] [2020-05-08 10:54:54,162 f5_cccl.resource.resource INFO] Updating ApiInternalDataGroup: /<PARTITION>/https_redirect_dg
2020/05/08 10:54:54 [ERROR] [2020-05-08 10:54:54,437 f5_cccl.resource.resource ERROR] HTTP error(400): CCCL resource(ApiInternalDataGroup) /<PARTITION>/https_redirect_dg.
2020/05/08 10:54:54 [ERROR] [2020-05-08 10:54:54,437 f5_cccl.service.manager ERROR] F5CcclResourceRequestError - 400 Unexpected Error: Bad Request for uri: https://F5_IP_ADDRESS:443/mgmt/tm/ltm/data-group/internal/~<PARTITION>~https_redirect_dg/
2020/05/08 10:54:54 [INFO] Text: u'{"code":400,"message":"0107074b:3: Unable to change data group (/<PARTITION>/https_redirect_dg) type.  Must remove existing entries first.","errorStack":[],"apiError":3}'
2020/05/08 10:54:54 [ERROR] [2020-05-08 10:54:54,438 f5_cccl.service.manager ERROR] Resource /<PARTITION>/https_redirect_dg update error, requeuing task...
2020/05/08 10:54:54 [ERROR] [2020-05-08 10:54:54,746 __main__ ERROR] Error applying config, will try again in 2 seconds

The ingress resources are separate manifests. Each manifest looks like the following

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: <NAME>
  namespace: <NAMESPACE>
  annotations:
    virtual-server.f5.com/ip: "controller-default"
    virtual-server.f5.com/partition: "<PARTITION>"
    virtual-server.f5.com/balance: "round-robin"
    ingress.kubernetes.io/allow-http: "false"
    ingress.kubernetes.io/ssl-redirect: "true"
    virtual-server.f5.com/health: |
      [
        {
          "path":     "<PATH>/",
          "send":     "HTTP GET /",
          "interval": 5,
          "timeout":  10
        }
      ]
    kubernetes.io/ingress.class: "f5"
spec:
  tls:
    - secretName: <SECRET_NAME
  rules:
    - host: <PATH>
      http:
        paths:
          - backend:
              serviceName: <SERVICE_NAME>
              servicePort: 9100
            path: /

Using the controller with our BigIP appliance is currently not working either. According to #975, the issue said it is fixed however it isn’t.