franz-go: Deadlock producing during shutdown

We have observed what looks like a deadlock in franz-go. We are using the ProduceSync method, and when we cancel the passed-in context, the call blocks forever.

Client configuration:

kgo.DisableIdempotentWrite(),
kgo.SeedBrokers(bootstrapAddrs),
kgo.WithLogger(kgoLogAdapter{ctx: ctx}),
kgo.RecordPartitioner(newKgoChangefeedPartitioner()),
kgo.ProducerBatchMaxBytes(256 << 20), // 256MiB
kgo.BrokerMaxWriteBytes(1 << 30),     // 1GiB
kgo.AllowAutoTopicCreation(),
kgo.RecordRetries(5),
kgo.RequestRetries(5),
kgo.ProducerOnDataLossDetected(func(topic string, part int32) {
        log.Errorf(ctx, `kafka sink detected data loss for topic %s partition %d`, redact.SafeString(topic), redact.SafeInt(part))
}),

kgo version: v1.17.1

Of note is that the client seems to be busy looping attempting to connect to the kafka broker, which i guess went away:

// kgo is spamming the logs with this at like 4k logs/hr (only ~1/sec but still)
WARN unable to open connection to broker ‹addr›=‹redacted.servicebus.windows.net:9093› ‹broker›=‹seed_0› ‹err›=dial tcp: ‹lookup redacted.servicebus.windows.net on 127.0.0.53:53: no such host›
INFO metadata update triggered ‹why›=‹re-updating metadata due to err: unable to dial: dial tcp: lookup redacted.servicebus.windows.net on 127.0.0.53:53: no such host›
WARN unable to open connection to broker ‹addr›=‹redacted.servicebus.windows.net:9093› ‹broker›=‹0› ‹err›=dial tcp: ‹lookup cdcdrt.servicebus.windows.net on 127.0.0.53:53: no such host›

Some relevant call stacks:

// (1) ProduceSync call being blocked on waiting for the individual produce calls to finish
1 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d46c5 0x4e6348 0x4607d2f 0x48bf9a6 0x48bf8db 0x486608b 0x489e12a 0xf07f73 0xf07d06 0xf07ebe 0x489df9d 0x489dc14 0x489db3f 0x1cdf7f6 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
#	0x4d46c4	sync.runtime_Semacquire+0x24										GOROOT/src/runtime/sema.go:62
#	0x4e6347	sync.(*WaitGroup).Wait+0x47										GOROOT/src/sync/waitgroup.go:116
#	0x4607d2e	github.com/twmb/franz-go/pkg/kgo.(*Client).ProduceSync+0x14e						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:263
#	0x48bf9a5	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush.func1+0x65		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:155
#	0x48bf8da	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush+0x9a			github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:181
#	0x486608a	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*batchingSink).runBatchingWorker.func1+0xea	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/batching_sink.go:353
#	0x489e129	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1.1+0x129		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:265
#	0xf07f72	github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1+0x12				github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:200
#	0xf07d05	github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do+0x245					github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:177
#	0xf07ebd	github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts+0xdd					github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:199
#	0x489df9c	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1+0x1dc		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:260
#	0x489dc13	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2+0xb3		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:277
#	0x489db3e	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6+0x1e	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
#	0x1cdf7f5	golang.org/x/sync/errgroup.(*Group).Go.func1+0x55							golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78


// (59) stuck in condvar wait awaken
59 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x4e4872 0x4e2cf7 0x4609149 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
#	0x4d4784	sync.runtime_SemacquireMutex+0x24				GOROOT/src/runtime/sema.go:77
#	0x4e4a3c	sync.(*Mutex).lockSlow+0x15c					GOROOT/src/sync/mutex.go:171
#	0x4e4871	sync.(*Mutex).Lock+0x31						GOROOT/src/sync/mutex.go:90
#	0x4e2cf6	sync.(*Cond).Wait+0x96						GOROOT/src/sync/cond.go:71
#	0x4609148	github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func2+0xc8	github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:475


// (43) stuck in <-wait
43 @ 0x4a014e 0x468ddf 0x4689f2 0x4608f31 0x4608b54 0x4607d0e 0x4607cef 0x48bf9a6 0x48bf8db 0x486608b 0x489e12a 0xf07f73 0xf07d06 0xf07ebe 0x489df9d 0x489dc14 0x489db3f 0x1cdf7f6 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
#	0x4608f30	github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func3+0x110						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:487
#	0x4608b53	github.com/twmb/franz-go/pkg/kgo.(*Client).produce+0xb73						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:500
#	0x4607d0d	github.com/twmb/franz-go/pkg/kgo.(*Client).Produce+0x12d						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:379
#	0x4607cee	github.com/twmb/franz-go/pkg/kgo.(*Client).ProduceSync+0x10e						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:261
#	0x48bf9a5	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush.func1+0x65		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:155
#	0x48bf8da	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush+0x9a			github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:181
#	0x486608a	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*batchingSink).runBatchingWorker.func1+0xea	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/batching_sink.go:353
#	0x489e129	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1.1+0x129		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:265
#	0xf07f72	github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1+0x12				github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:200
#	0xf07d05	github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do+0x245					github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:177
#	0xf07ebd	github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts+0xdd					github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:199
#	0x489df9c	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1+0x1dc		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:260
#	0x489dc13	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2+0xb3		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:277
#	0x489db3e	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6+0x1e	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
#	0x1cdf7f5	golang.org/x/sync/errgroup.(*Group).Go.func1+0x55							golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78

// (30) stuck in mu.Lock() in drainbuffered?
30 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x4608eb7 0x4608e85 0x4608b54 0x4607d0e 0x4607cef 0x48bf9a6 0x48bf8db 0x486608b 0x489e12a 0xf07f73 0xf07d06 0xf07ebe 0x489df9d 0x489dc14 0x489db3f 0x1cdf7f6 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
#	0x4d4784	sync.runtime_SemacquireMutex+0x24									GOROOT/src/runtime/sema.go:77
#	0x4e4a3c	sync.(*Mutex).lockSlow+0x15c										GOROOT/src/sync/mutex.go:171
#	0x4608eb6	sync.(*Mutex).Lock+0x96											GOROOT/src/sync/mutex.go:90
#	0x4608e84	github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func3+0x64						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:483
#	0x4608b53	github.com/twmb/franz-go/pkg/kgo.(*Client).produce+0xb73						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:500
#	0x4607d0d	github.com/twmb/franz-go/pkg/kgo.(*Client).Produce+0x12d						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:379
#	0x4607cee	github.com/twmb/franz-go/pkg/kgo.(*Client).ProduceSync+0x10e						github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:261
#	0x48bf9a5	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush.func1+0x65		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:155
#	0x48bf8da	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush+0x9a			github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:181
#	0x486608a	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*batchingSink).runBatchingWorker.func1+0xea	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/batching_sink.go:353
#	0x489e129	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1.1+0x129		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:265
#	0xf07f72	github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1+0x12				github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:200
#	0xf07d05	github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do+0x245					github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:177
#	0xf07ebd	github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts+0xdd					github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:199
#	0x489df9c	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1+0x1dc		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:260
#	0x489dc13	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2+0xb3		github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:277
#	0x489db3e	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6+0x1e	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
#	0x1cdf7f5	golang.org/x/sync/errgroup.(*Group).Go.func1+0x55							golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78


// (13) stuck in mu.Lock() in the goroutine?
13 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x460911a 0x46090ff 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
#	0x4d4784	sync.runtime_SemacquireMutex+0x24				GOROOT/src/runtime/sema.go:77
#	0x4e4a3c	sync.(*Mutex).lockSlow+0x15c					GOROOT/src/sync/mutex.go:171
#	0x4609119	sync.(*Mutex).Lock+0x99						GOROOT/src/sync/mutex.go:90
#	0x46090fe	github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func2+0x7e	github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:472


// (1) waiting on lock in finishing promises? not sure
1 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x46098fd 0x46098ca 0x460971f 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
#	0x4d4784	sync.runtime_SemacquireMutex+0x24					GOROOT/src/runtime/sema.go:77
#	0x4e4a3c	sync.(*Mutex).lockSlow+0x15c						GOROOT/src/sync/mutex.go:171
#	0x46098fc	sync.(*Mutex).Lock+0x13c						GOROOT/src/sync/mutex.go:90
#	0x46098c9	github.com/twmb/franz-go/pkg/kgo.(*Client).finishRecordPromise+0x109	github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:587
#	0x460971e	github.com/twmb/franz-go/pkg/kgo.(*producer).finishPromises+0x2be	github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:549

I’ll keep looking at this but your assistance would be greatly appreciated

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

ETA today. I’ll reopen and then close one I push.

Merging my PR, but await the tag.

Standard case, overMaxRecs || overMaxBytes becomes false due to a record finishing.