argo-events: Gateway pod is leaking memory.

Here’s my update:

• Migrating from 0.11 to 0.12 was terrible – no docs and a ton of model changes. Thankfully, the changes were fairly good and well structured, so 🙌 for that!

• 0.12 seems to have fixed the memory leaks 🎉 🎉 🥂

Here are the notes I took while migrating, and a summary of changes I noticed:

• Migrating from 0.11 to 0.12 webhook configs:

  • Webhook Gateway

    • “watchers” turned into “subscribes” and points to the service endpoint, not the sensor itself

  • Webhook Sensor

    • dependencies specify the gateway in a formalized object of { name, gatewayName, eventName }, EX: old format of my-gateway:my-event-source is now { name: 'myEventName', gatewayName: 'my-gateway', eventName: 'my-event-source' }, you can access this in parameters using myEventName.
    • Change eventProtocol to subscription object to attach to the gateway
    • Major changes/formalization of parameterization – a little more nested and mostly more consistent.
    • MAJOR gotcha around what object the parameters apply to. parameters as a child of template see the entire template object. parameters as a child of the k8s object in the template see k8s.source as the object – so, one extra layer deep. Example below:

      triggers:
        - template:
            # all these spaces in is correct
            name: my-webhook-workflow
            k8s:
              group: argoproj.io
              version: v1alpha1
              resource: workflows
              operation: create
              source:
                git:
                  url: "git@github.com:some-github/repo.git"
                  cloneDirectory: "/git"
                  sshKeyPath: "/secret/key"
                  filePath: "TEMPLATED_FILLS_IN"
                  branch: "master"
    
              # Resource-level paramters - the paths here are rooted at the !!! k8s.source object !!!
              parameters:
                - src:
                    dependencyName: gateway-dep
                    dataKey: body.some.other.path
                  # Notice it is NOT source.spec or k8s.source, but spec.
                  dest: spec.arguments.parameters.0.value
          # Trigger-level parameters -- the paths here are rooted at the entire template object.
          parameters:
            - src:
                dependencyName: gateway-dep
          # dataKey will deserialize the data base64, it is a json object of { body, headers, ... }
                dataKey: body.some.json.path
                value: "master"
          # Notice k8s pathing
              dest: k8s.source.git.branch
    

    • EventSoruce
      • Actual custom resource as opposed to a config map
      • Formalized the document – spec types define the event source shape (IE: webhook) and the further configs, the example for a webhook event source is easy enough to follow.

Other notes:

• Triggers sources are standardized until k8s standard resources, so they’re all nested 1 more deep
• Event source dependencies are formalized, you declare the dependency then use it in params
• Parameterization is formalized as a dependency source and then a contextKey or a dataKey
  • Events have { context, data } shape where context is meta about the event itself.
  • At the event level, data is base64 encoded raw of whatever the payload is
  • Depending on the event, data can take different forms/values
  • For webhooks, data is the JSON of the full http request, IE: { header, body }
  • To extract webhook payload json, you need to do: dataKey: body.[...], like dataKey: body.some.json.path
• Be careful about the nesting for parameters in a trigger definition – where is important.
  • https://github.com/argoproj/argo-events/blob/a069dd836dcab43f09b5a1ebf1a9a6d56d897e26/sensors/event-notifications.go#L75-L128
  • The general flow is:
    • Apply template level parameters
      • template.parameters to template document
      • IE: k8s.source.hit as the dest
    • Check if switches
    • Fetch resource
      • This is where k8s.source gets resolved/built
    • apply resource parmeters
      • template.k8s.parameters gets applied to template.k8s.source object
      • IE: spec.arguments.parameters.0
      • So, it doesn’t see “source” or other siblings of the k8s document
    • execute trigger

@emilssolmanis It’s been 12 days since your comment. Did the upgrade fix the leak? Did you upgrade from 0.10? Any tips when upgrading?

This is definitely here on v0.11, had it die after ~15 days or so.