enqueue-dev: Kafka: Offset commit (manual) failed

My consumer always fails to commit the offset:

%4|1560006005.731|COMMITFAIL|rdkafka#consumer-20| [thrd:main]: Offset commit (manual) failed for 1/1 partition(s): Broker: Unknown member: commands[0]@1(Broker: Unknown member)

Here is my Config:

        consumer:
            dsn: "..."
            global:
                ### Make sure this is unique for each application / consumer group and does not change
                ### Otherwise, Kafka won't be able to track your last offset and will always start according to
                ### `auto.offset.reset` setting.
                ### See Kafka documentation regarding `group.id` property if you want to know more
                group.id: 'backend'
                metadata.broker.list: '...'
                security.protocol: sasl_ssl
                ssl.ca.location: '%kernel.project_dir%/config/kafka/ca.pem'
                sasl.username: '...'
                sasl.password: '...'
                sasl.mechanisms: "SCRAM-SHA-256"
                max.poll.interval.ms: '25000'
                enable.auto.commit: 'false'
                enable.auto.offset.store: 'false'
            topic:
                auto.offset.reset: beginning
            ### Commit async is true by default since version 0.9.9.
            ### It is suggested to set it to true in earlier versions since otherwise consumers become extremely slow,
            ### waiting for offset to be stored on Kafka before continuing.
            commit_async: true

Any ideas, why this is happening?

Tested with both heroku Kafka & CloudKarafka.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (9 by maintainers)

Most upvoted comments

@marcoreni just to let you know I’ll have difficulty having a closer look on it before saturday, 07.09. I’m keeping it in the back of my head.

Steveb-p on Sep 5, 2019

Whoopsie, sorry @marcoreni 😄

Steveb-p on Nov 21, 2019

@KonstantinCodes @Steveb-p seeing the same issue after upgrading to Symfony 4.3 (and sroze/enqueue-messenger-transport 0.3.2). Everything was working fine before (enqueue 0.9.12, symfony 4.2.*, sroze/enqueue-messenger-transport 0.2.2) .

Maybe this is due to sroze/messenger-enqueue-transport#75 ?

Hi, the two issues #893 and #894 already existed before I made the PR. The PR basically just fixed the issue that the offset value was getting lost.

KonstantinCodes on Sep 3, 2019

https://github.com/marcoreni/enqueue-kafka-894-repro

Here you go. I tried to keep it as simple as possible, I just added two docker containers to make testing easier. It includes a command to dispatch the message, and a script to dispatch a new message every 5 seconds.

My latest test was able to handle two messages (even if the COMMITFAIL was returned even for the first message:

php_1    | 2019-09-03T09:03:18+00:00 [info] Received message App\Message\TestMessage
php_1    | 2019-09-03T09:03:18+00:00 [info] Received a new message.
php_1    | 2019-09-03T09:03:18+00:00 [info] Message App\Message\TestMessage handled by App\MessageHandler\TestMessageHandler::__invoke
php_1    | 2019-09-03T09:03:18+00:00 [info] App\Message\TestMessage was handled successfully (acknowledging to transport).
php_1    | %4|1567501400.934|COMMITFAIL|rdkafka#consumer-2| [thrd:main]: Offset commit (manual) failed for 1/1 partition(s): Broker: Unknown member: test[0]@4(Broker: Unknown member)
php_1    | 2019-09-03T09:03:24+00:00 [info] Received message App\Message\TestMessage
php_1    | 2019-09-03T09:03:24+00:00 [info] Received a new message.
php_1    | 2019-09-03T09:03:24+00:00 [info] Message App\Message\TestMessage handled by App\MessageHandler\TestMessageHandler::__invoke
php_1    | 2019-09-03T09:03:24+00:00 [info] App\Message\TestMessage was handled successfully (acknowledging to transport).
php_1    | %4|1567501405.934|COMMITFAIL|rdkafka#consumer-4| [thrd:main]: Offset commit (manual) failed for 1/1 partition(s): Broker: Unknown member: test[0]@5(Broker: Unknown member)

Since it’s a newly created project I’m using latest version for each package.

(BTW, with this same repository you are also able to reproduce #893. You simply have to stop dispatching messages and wait 😃 )

marcoreni on Sep 3, 2019