k8s-config-connector: GCP resources created by KCC does not allow for dynamic fields

Use case: Use KCC to create GCP resources used as Traffic Director configuration

We create GCP resources using KCC

healthcheck
backend service
urlmap …

We annotate k8s service object with cloud.google.com/neg and anthos.cft.dev/autoneg which will respectively create NEGs, and add them to the backend service.
These NEGs are removed from the backend service by the KCC controller, which is not the desired behavior for us.

Tried to workaround by

create healtcheck with KCC
create backend service with gcloud command
create urlmap and other resources with KCC

This makes us run into issue #66, that we can’t create the resources referring to the backend service. We tried to workaround this by first creating the backend service with gcloud and then creating the KCC resource for it, this allows for creating the urlmaps etc, but then we run into the same problem we started with, KCC deletes the NEGs from the backend service.

Would it be possible to annotate the KCC resources with some GCP resource fields that KCC will not touch?

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 22 (12 by maintainers)

Most upvoted comments

We are planning to support Server Side apply which should make it possible for customers of KCC to specify values for fields they want to manage. For fields they do not specify, KCC will allow “GCP” to be the source of truth. With that information, KCC will be able to sync the state of the BackendService into the API server and know to NOT overwrite the NEG list.

https://kubernetes.io/docs/reference/using-api/api-concepts/#server-side-apply

spew on Feb 3, 2020

Hi. I am hitting this issue too.

I have a different idea for a solution however (let me know what you think):

Currently the name of the NEG (Network Endpoint Group) that is created automatically when you define a Service is based on the service’s name and port and ends with a random string - so isn’t knowable in advance - which makes configuration as code difficult.

Image instead that you could configure the name of the NEG in the Service yaml. For example: "name": "my-neg".

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: my-project
  annotations:
    cloud.google.com/neg: '{"exposed_ports":{"5990":{}, "8080":{}}, "name": "my-neg"}'

With this approach, you could configure a BackendService backed by NEGs running in one or more GKE clusters without using the anthos autoneg controller.

You would instead define a ComputeBackendService and ComputeNetworkEndpointGroup using KCC as follows:

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeBackendService
metadata:
  name: my-backend-service
  namespace: my-project
spec:
  affinityCookieTtlSec: 0
  backend:
    - balancingMode: RATE
      capacityScaler: 1.0
      maxRatePerEndpoint: 1000.0
      group:
        networkEndpointGroupRef:
          name: my-neg # reference the NEG defined below
    # optionally add other backends here, eg from another zone in the cluster, or even from another GKE cluster

---

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeNetworkEndpointGroup
metadata:
  name: my-neg # this matches the name defined in the Service annotation above
  namespace: my-project
spec:
  networkEndpointType: GCE_VM_IP_PORT
  networkRef:
    ...
  subnetworkRef:
    ...
  location: ...

This would I imagine need to be filed as an feature request against the GCE Ingress repo.

mikelnrd on Mar 3, 2020

@omaskery, the best way to track the progress of this feature is through this issue thread. As of right now, we are actively working on this feature and are aiming to get a solution out within this quarter/in the next few months.

caieo on Apr 22, 2020