cilium: Cilium upstream master branch breaks nodeport connection from external client

Bug report

Cilium upstream master branch breaks nodeport connection from external client,

General Information

Cilium version (run cilium version) Client: 1.10.90 c44ff1b37 2021-08-03T00:35:29+05:30 go version go1.16.5 linux/amd64 Daemon: 1.10.90 c44ff1b37 2021-08-03T00:35:29+05:30 go version go1.16.5 linux/amd64
Kernel version (run uname -a) 5.8.1-050801-generic #202008111432 SMP Tue Aug 11 14:34:42 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Orchestration system version in use (e.g. kubectl version, …)

Client Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.5”, GitCommit:“6b1d87acf3c8253c123756b9e61dac642678305f”, GitTreeState:“clean”, BuildDate:“2021-03-18T01:10:43Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.8”, GitCommit:“5575935422cc1cf5169dfc8847cb587aa47bac5a”, GitTreeState:“clean”, BuildDate:“2021-06-16T12:53:07Z”, GoVersion:“go1.15.13”, Compiler:“gc”, Platform:“linux/amd64”}

Link to relevant artifacts (policies, deployments scripts, …)

cat nginx_nodeport.yaml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 1 
  selector:
    app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---

apiVersion: v1
kind: Service
metadata:
  labels:
    name: nginxservice
  name: nginxservice
spec:
  ports:
    # The port that this service should serve on.
    - port: 80
      nodePort: 32506
  selector:
    app: nginx
  type: NodePort

Generate and upload a system zip:

can upload if needed

How to reproduce the issue

instruction 1

build docker image from upstream master branch

instruction

deploy cilium with attached cilium yaml file

instruction

deploy above nodeport service

5, access the nodeport service from external client

cilium-upstream.yaml.txt

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 39 (39 by maintainers)

Most upvoted comments

@Weil0ng yes, make docker-operator-generic-image and run the operator image fixed the problem, sorry I did not realize the operator image better match the agent image build 😃

vincentmli on Oct 29, 2021

yes, the problem still here

kubectl get crd ciliumendpoints.cilium.io -o yaml


apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2021-06-14T22:12:59Z"
  generation: 1
  labels:
    io.cilium.k8s.crd.schema.version: 1.23.3
  name: ciliumendpoints.cilium.io
  resourceVersion: "13829675"
  uid: 0ef89eac-8dbd-488a-bbc5-cd8a00918c8d
spec:
  conversion:
    strategy: None
  group: cilium.io
  names:
    kind: CiliumEndpoint
    listKind: CiliumEndpointList
    plural: ciliumendpoints
    shortNames:
    - cep
    - ciliumep
    singular: ciliumendpoint
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Cilium endpoint id
      jsonPath: .status.id
      name: Endpoint ID
      type: integer
    - description: Cilium identity id
      jsonPath: .status.identity.id
      name: Identity ID
      type: integer
    - description: Ingress enforcement in the endpoint
      jsonPath: .status.policy.ingress.enforcing
      name: Ingress Enforcement
      type: boolean
    - description: Egress enforcement in the endpoint
      jsonPath: .status.policy.egress.enforcing
      name: Egress Enforcement
      type: boolean
    - description: Status of visibility policy in the endpoint
      jsonPath: .status.visibility-policy-status
      name: Visibility Policy
      type: string
    - description: Endpoint current state
      jsonPath: .status.state
      name: Endpoint State
      type: string
    - description: Endpoint IPv4 address
      jsonPath: .status.networking.addressing[0].ipv4
      name: IPv4
      type: string
    - description: Endpoint IPv6 address
      jsonPath: .status.networking.addressing[0].ipv6
      name: IPv6
      type: string
    name: v2
    schema:
      openAPIV3Schema:
        description: CiliumEndpoint is the status of a Cilium policy rule.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          status:
            description: EndpointStatus is the status of a Cilium endpoint.
            properties:
              controllers:
                description: Controllers is the list of failing controllers for this
                  endpoint.
                items:
                  description: ControllerStatus is the status of a failing controller.
                  properties:
                    configuration:
                      description: Configuration is the controller configuration
                      properties:
                        error-retry:
                          description: Retry on error
                          type: boolean
                        error-retry-base:
                          description: 'Base error retry back-off time Format: duration'
                          format: int64
                          type: integer
                        interval:
                          description: 'Regular synchronization interval Format: duration'
                          format: int64
                          type: integer
                      type: object
                    name:
                      description: Name is the name of the controller
                      type: string
                    status:
                      description: Status is the status of the controller
                      properties:
                        consecutive-failure-count:
                          format: int64
                          type: integer
                        failure-count:
                          format: int64
                          type: integer
                        last-failure-msg:
                          type: string
                        last-failure-timestamp:
                          type: string
                        last-success-timestamp:
                          type: string
                        success-count:
                          format: int64
                          type: integer
                      type: object
                    uuid:
                      description: UUID is the UUID of the controller
                      type: string
                  type: object
                type: array
              encryption:
                description: Encryption is the encryption configuration of the node
                properties:
                  key:
                    description: Key is the index to the key to use for encryption
                      or 0 if encryption is disabled.
                    type: integer
                type: object
              external-identifiers:
                description: ExternalIdentifiers is a set of identifiers to identify
                  the endpoint apart from the pod name. This includes container runtime
                  IDs.
                properties:
                  container-id:
                    description: ID assigned by container runtime
                    type: string
                  container-name:
                    description: Name assigned to container
                    type: string
                  docker-endpoint-id:
                    description: Docker endpoint ID
                    type: string
                  docker-network-id:
                    description: Docker network ID
                    type: string
                  k8s-namespace:
                    description: K8s namespace for this endpoint
                    type: string
                  k8s-pod-name:
                    description: K8s pod name for this endpoint
                    type: string
                  pod-name:
                    description: K8s pod for this endpoint(Deprecated, use K8sPodName
                      and K8sNamespace instead)
                    type: string
                type: object
              health:
                description: Health is the overall endpoint & subcomponent health.
                properties:
                  bpf:
                    description: bpf
                    type: string
                  connected:
                    description: Is this endpoint reachable
                    type: boolean
                  overallHealth:
                    description: overall health
                    type: string
                  policy:
                    description: policy
                    type: string
                type: object
              id:
                description: ID is the cilium-agent-local ID of the endpoint.
                format: int64
                type: integer
              identity:
                description: Identity is the security identity associated with the
                  endpoint
                properties:
                  id:
                    description: ID is the numeric identity of the endpoint
                    format: int64
                    type: integer
                  labels:
                    description: Labels is the list of labels associated with the
                      identity
                    items:
                      type: string
                    type: array
                type: object
              log:
                description: Log is the list of the last few warning and error log
                  entries
                items:
                  description: "EndpointStatusChange Indication of a change of status
                    \n swagger:model EndpointStatusChange"
                  properties:
                    code:
                      description: 'Code indicate type of status change Enum: [ok
                        failed]'
                      type: string
                    message:
                      description: Status message
                      type: string
                    state:
                      description: state
                      type: string
                    timestamp:
                      description: Timestamp when status change occurred
                      type: string
                  type: object
                type: array
              named-ports:
                description: "NamedPorts List of named Layer 4 port and protocol pairs
                  which will be used in Network Policy specs. \n swagger:model NamedPorts"
                items:
                  description: "Port Layer 4 port / protocol pair \n swagger:model
                    Port"
                  properties:
                    name:
                      description: Optional layer 4 port name
                      type: string
                    port:
                      description: Layer 4 port number
                      type: integer
                    protocol:
                      description: 'Layer 4 protocol Enum: [TCP UDP ANY]'
                      type: string
                  type: object
                type: array
              networking:
                description: Networking is the networking properties of the endpoint.
                properties:
                  addressing:
                    description: IP4/6 addresses assigned to this Endpoint
                    items:
                      description: AddressPair is is a par of IPv4 and/or IPv6 address.
                      properties:
                        ipv4:
                          type: string
                        ipv6:
                          type: string
                      type: object
                    type: array
                  node:
                    description: NodeIP is the IP of the node the endpoint is running
                      on. The IP must be reachable between nodes.
                    type: string
                required:
                - addressing
                type: object
              policy:
                description: EndpointPolicy represents the endpoint's policy by listing
                  all allowed ingress and egress identities in combination with L4
                  port and protocol.
                properties:
                  egress:
                    description: EndpointPolicyDirection is the list of allowed identities
                      per direction.
                    properties:
                      adding:
                        description: Deprecated
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                      allowed:
                        description: AllowedIdentityList is a list of IdentityTuples
                          that species peers that are allowed.
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                      denied:
                        description: DenyIdentityList is a list of IdentityTuples
                          that species peers that are denied.
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                      enforcing:
                        type: boolean
                      removing:
                        description: Deprecated
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                    required:
                    - enforcing
                    type: object
                  ingress:
                    description: EndpointPolicyDirection is the list of allowed identities
                      per direction.
                    properties:
                      adding:
                        description: Deprecated
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                      allowed:
                        description: AllowedIdentityList is a list of IdentityTuples
                          that species peers that are allowed.
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                      denied:
                        description: DenyIdentityList is a list of IdentityTuples
                          that species peers that are denied.
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                      enforcing:
                        type: boolean
                      removing:
                        description: Deprecated
                        items:
                          description: IdentityTuple specifies a peer by identity,
                            destination port and protocol.
                          properties:
                            dest-port:
                              type: integer
                            identity:
                              format: int64
                              type: integer
                            identity-labels:
                              additionalProperties:
                                type: string
                              type: object
                            protocol:
                              type: integer
                          type: object
                        type: array
                    required:
                    - enforcing
                    type: object
                type: object
              state:
                description: State is the state of the endpoint.
                enum:
                - creating
                - waiting-for-identity
                - not-ready
                - waiting-to-regenerate
                - regenerating
                - restoring
                - ready
                - disconnecting
                - disconnected
                - invalid
                type: string
              visibility-policy-status:
                type: string
            type: object
        required:
        - metadata
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: CiliumEndpoint
    listKind: CiliumEndpointList
    plural: ciliumendpoints
    shortNames:
    - cep
    - ciliumep
    singular: ciliumendpoint
  conditions:
  - lastTransitionTime: "2021-06-14T22:12:59Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2021-06-14T22:12:59Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v2

vincentmli on Oct 28, 2021

I think so, the CRDs are registered by the operator, if it does not restart, I suspect the CRDs were never updated. So eventually you have an outdated CRD (which has status as a subresource) with the new agent code which will update the CEP assuming it is using the new CRD (which has status as a plain field), and that’s why we see the object fully populated in the API request but not persisting in the etcd.

Weil0ng on Oct 29, 2021