jaeger: es-index-cleaner:latest is not working as expected

@ichasco-heytrade here is my reproducer for https://github.com/jaegertracing/jaeger/issues/3228#issuecomment-918217814 to remove jaeger-jaeger-dependencies-2021-09-10. The test passes so it means that the index will be deleted.

PR https://github.com/jaegertracing/jaeger/pull/3264

To remove the jaeger-jaeger-dependencies-2021-09-10 try running:

docker run --rm -it -e INDEX_PREFIX=jaeger pavolloffay/jaeger-es-index-cleaner:PR3261 3 <es-url>

Do not enable rollover bc the index is not managed by the rollover.

It seems that image jaegertracing/jaeger-es-index-cleaner:1 16 days ago was jaegertracing/jaeger-es-index-cleaner@sha256:5e0adb0e8623dd334709943999a6b9d3e283090efcc05a7f90c34804a7e71758 and it was with python implementation - it save us that moment.

I check pod with image jaeger-es-index-cleaner:1 today - and now this image is in go implemetation

jaegertracing/jaeger-es-index-cleaner@sha256:520279aa82234dd08e3439907e2ec229c6501524a7886f133fd7c897c86667c8

todays logs of jaeger-es-index-cleaner pod is No indices to delete

{"level":"info","ts":1631174408.3293278,"caller":"./main.go:83","msg":"Indices before this date will be deleted","date":1631145600}
{"level":"info","ts":1631174408.3294582,"caller":"./main.go:95","msg":"No indices to delete"}

@pavolloffay I think the problem in environment

env:
- name: ES_INDEX_PREFIX
   value: jaeger-prod

it should be

env:
- name: INDEX_PREFIX
   value: jaeger-prod

But I do not understand how it can work in python implementation early…

Today I try to run it with

kubectl run test --image jaegertracing/jaeger-es-index-cleaner@sha256:5e0adb0e8623dd334709943999a6b9d3e283090efcc05a7f90c34804a7e71758 --restart Never -it --rm --command sh

# and run
export ES_TLS_ENABLED="true"
export ES_TLS_SKIP_HOST_VERIFY="true"
export ROLLOVER="true"
export ES_SERVER_URLS="http://elasticsearch-master.elk.svc.cluster.local:9200"
export ES_USERNAME="test"
export ES_PASSWORD="test"
export ES_INDEX_PREFIX="jaeger-prod"

python3 /es-index-cleaner/esCleaner.py "1" http://elasticsearch-master.elk.svc.cluster.local:9200

result

usr/local/lib/python3.8/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .kibana_7.12.0_001, .kibana_task_manager_7.12.0_001, .tasks], but in a future major version, direct access to system indices will be prevented by default
  warnings.warn(message, category=ElasticsearchWarning)
Removing jaeger-prod-jaeger-span-001027
Removing jaeger-prod-jaeger-span-001025
Removing jaeger-prod-jaeger-span-001022
Removing jaeger-prod-jaeger-span-001031
Removing jaeger-prod-jaeger-service-000993
Removing jaeger-prod-jaeger-service-000994
Removing jaeger-prod-jaeger-span-001030
Removing jaeger-prod-jaeger-service-000989
Removing jaeger-prod-jaeger-service-000998
Removing jaeger-prod-jaeger-span-001028
Removing jaeger-prod-jaeger-service-000997
Removing jaeger-prod-jaeger-span-001029
Removing jaeger-prod-jaeger-span-001020
Removing jaeger-prod-jaeger-span-001023
Removing jaeger-prod-jaeger-service-000990
Removing jaeger-prod-jaeger-span-001024
Removing jaeger-prod-jaeger-service-000987
Removing jaeger-prod-jaeger-span-001021
Removing jaeger-prod-jaeger-service-000988
Removing jaeger-prod-jaeger-service-000992
Removing jaeger-prod-jaeger-service-000995
Removing jaeger-prod-jaeger-service-000996
Removing jaeger-prod-jaeger-service-000991
Removing jaeger-prod-jaeger-span-001026

with go implementation, with this environment - No indices to delete

{"level":"info","ts":1631174408.3293278,"caller":"./main.go:83","msg":"Indices before this date will be deleted","date":1631145600}
{"level":"info","ts":1631174408.3294582,"caller":"./main.go:95","msg":"No indices to delete"}

@pavolloffay Sorry for this noise, it seems some elasticsearch specific… It’s not a bug for go implementation

Sure. As I mension before - we use jaeger installation in kubernetes with elasticsearch backend and rollover - we install jaeger with helm chart with enabled esIndexCleaner to clear old indexes in our elasticsearch backend to reduce disk usage

...
esIndexCleaner:
  enabled: true
  schedule: "0 * * * *"
  numberOfDays: 1
  extraEnv:
  - name: ES_TLS_ENABLED
    value: "true"
  - name: ES_TLS_SKIP_HOST_VERIFY
    value: "true"
  - name: ROLLOVER
    value: 'true'
...

It’s create CronJob with this specification

...
spec:
  containers:
  - args:
    - "1"
    - http://elasticsearch:9200
    env:
    - name: ES_TLS_ENABLED
      value: "true"
    - name: ES_TLS_SKIP_HOST_VERIFY
      value: "true"
    - name: ROLLOVER
      value: "true"
    - name: ES_SERVER_URLS
      value: http://elasticsearch:9200
    - name: ES_USERNAME
      value: elastic
    - name: ES_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: jaeger-elasticsearch
    - name: ES_INDEX_PREFIX
      value: jaeger-prod
    image: jaegertracing/jaeger-es-index-cleaner:latest
...

It’s successfuly works until today - we notice that our disk usage significantly increased - we realize that old jaeger indexes not removing - cronjob logs

{"level":"info","ts":1629795860.802725,"caller":"./main.go:83","msg":"Indices before this date will be deleted","date":1629763200}
{"level":"info","ts":1629795860.8028526,"caller":"./main.go:95","msg":"No indices to delete"}

I think that the problem in es-index-cleaner golang implementation (#3192) this implementation do not use environment variables that was used in old python implementation - we clear old indexes only with dowgrading to python implementation jaegertracing/jaeger-es-index-cleaner:1