connectors: ESET connector causing runaway Redis memory consumption despite REDIS__TRIMMING=1000000

Description

When enabling the external-import/eset connector with defaults (from external-import/eset/docker-compose.yml), it causes redis to rapidly run away with consuming all memory. I have REDIS__TRIMMING=1000000 set in my docker-compose.yml and it doesn’t seem to make a difference for this particular connector, but seems to be working for everything else.

The connector-eset section of my docker-compose.yml is lifted almost verbatim from the example provided in the connector’s directory:

  connector-eset:
    image: opencti/connector-eset:5.12.32
    environment:
      - OPENCTI_URL=http://opencti-url
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${ESET_CONNECTOR_ID}
      - "CONNECTOR_NAME=ESET Cyber Threat Intelligence"
      - CONNECTOR_SCOPE=eset
      - CONNECTOR_CONFIDENCE_LEVEL=30 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=error
      - ESET_API_URL=eti.eset.com
      - ESET_USERNAME=${ESET_CONNECTOR_USER}
      - ESET_PASSWORD=${ESET_CONNECTOR_PASSWORD}
      - ESET_COLLECTIONS=ei.misp,ei.cc,ei.urls
      - ESET_IMPORT_APT_REPORTS=true
      - ESET_IMPORT_START_DATE=2022-04-01
      - ESET_CREATE_OBSERVABLES=true
      - ESET_INTERVAL=60 # Required, in minutes
    restart: always

Environment

1. OS (where OpenCTI server runs): Amazon Linux 2023
2. OpenCTI version: 5.12.32
3. OpenCTI client: connector
4. Other environment details: Using Docker CE and docker-compose

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Add connector-eset in my docker-compose.yml
2. Update .env with proper credentials for access
3. docker-compose up -d
4. Wait for awhile and eventually redis grows to consume all RAM (in my case, it got to around 12GB in usage)

Expected Output

Redis shouldn’t consume all memory and REDIS__TRIMMING=.... should be able to keep it within a reasonably predictable ceiling. In this particular case, redis resident memory size seems to stay under 2GB when trimming is set to 1000000.

Actual Output

redis memory consumption grows without restraint until the system runs out of memory and the OOM reaper has to kill something.

Additional information

Here is a docker-compose.yml that seems to be working well for me with REDIS__TRIMMING=2000000: https://github.com/ckane/opencti-docker/blob/tf-main/docker-compose.yml

In the deployment I am trying to use it in, I reduced this to 1000000 to see if it would fix the problem, but it doesn’t seem to have any impact at all.

In this case, redis memory consumption stays under control, but if I add connector-eset to it, then when the eset connector starts ingesting intel from their feed, redis consumption will grow rapidly until it exhausts all RAM in the system.