franz-go: error="unknown attributes on uncompressed message 8"

We are piloting this project out and are running in to this error when trying to consume from one of our brokers:

unknown attributes on uncompressed message 8

The error is being returned from fetches.Errors()

This particular broker cluster has the following settings:

version=2.2.1
inter.broker.protocol.version: "2.2"
log.message.format.version: "0.10.1"

We do not see the same issue on another broker cluster with the following settings:

version: 2.7.1
inter.broker.protocol.version: 2.7.1
log.message.format.version: "2.7"

I’ve tried passing different versions to the client via kgo.MaxVersions to no avail

Is there some other option that I should be setting here or is there any way to get more debug information about what is happening here?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (10 by maintainers)

Most upvoted comments

Awesome, thank you! I’ve gone ahead and tagged this in v1.1.2. I’m going to close this for now but if anything else crops up please let me know!

twmb on Sep 28, 2021

Looking back at https://kafka.apache.org/documentation/#messageset,it appears I misunderstood their bits. I’m fairly sure I know the problem, and I can patch this in 20. I read bits 0 to 2 as the first two bits, not the first three, and that continued to make sense to me because Kafka never used more than the first two bits for compression. Thanks for bringing this up, I can patch this after lunch.

On Tue, Sep 28, 2021 at 13:16 Kevin Conaway @.***> wrote:

This https://github.com/twmb/franz-go/commit/6e639fb0844ee6c2cb385f15a9c70419cbe29844#diff-c6763587c65f4655271050afcde2ba38165b48756f75072ab54f3093246efc61R1281 fails in this little example https://play.golang.org/p/L7U_aHfKIJ1 as well

My bit twiddling skills leave a lot to be desired but can’t the 3rd bit for LogAppendTime be 0 or 1? If so, I think this code is only checking if its 0

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/twmb/franz-go/issues/90#issuecomment-929550084, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJZIJNLLY4EYT7FQW3YVPTUEIH75ANCNFSM5E57LYBA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

twmb on Sep 28, 2021

It looks like there is a compiler error here

That teaches me for thinking this was easy enough to not need to stash my other work, very sorry about that. I’ll run the integration tests for the next commit just to be sure…

I’m sorry, I don’t understand the question

The other cluster settings you posted show log.message.format.version: "2.7", which uses the “new” RecordBatch format, rather than the old MessageSet. The client isn’t as sensitive on attributes for the newer format. Point is, if the new format has LogAppendTime, that’ll be fine because the client doesn’t choke as hard there.

twmb on Sep 28, 2021

I think this is a tiny bug in this client. The broker is returning LogAppendTime (bit 3) for a message, and the client is not expecting that. This is almost certainly an error introduced in a previous large refactoring, and was uncaught because the integration tests are on new Kafkas (message sets were removed in Kafka 0.11).

I can patch this quickly and tag a patch release.

twmb on Sep 28, 2021