franz-go: Deadlock producing during shutdown
We have observed what looks like a deadlock in franz-go.
We are using the ProduceSync method, and when we cancel the passed-in context, the call blocks forever.
Client configuration:
kgo.DisableIdempotentWrite(),
kgo.SeedBrokers(bootstrapAddrs),
kgo.WithLogger(kgoLogAdapter{ctx: ctx}),
kgo.RecordPartitioner(newKgoChangefeedPartitioner()),
kgo.ProducerBatchMaxBytes(256 << 20), // 256MiB
kgo.BrokerMaxWriteBytes(1 << 30), // 1GiB
kgo.AllowAutoTopicCreation(),
kgo.RecordRetries(5),
kgo.RequestRetries(5),
kgo.ProducerOnDataLossDetected(func(topic string, part int32) {
log.Errorf(ctx, `kafka sink detected data loss for topic %s partition %d`, redact.SafeString(topic), redact.SafeInt(part))
}),
kgo version: v1.17.1
Of note is that the client seems to be busy looping attempting to connect to the kafka broker, which i guess went away:
// kgo is spamming the logs with this at like 4k logs/hr (only ~1/sec but still)
WARN unable to open connection to broker ‹addr›=‹redacted.servicebus.windows.net:9093› ‹broker›=‹seed_0› ‹err›=dial tcp: ‹lookup redacted.servicebus.windows.net on 127.0.0.53:53: no such host›
INFO metadata update triggered ‹why›=‹re-updating metadata due to err: unable to dial: dial tcp: lookup redacted.servicebus.windows.net on 127.0.0.53:53: no such host›
WARN unable to open connection to broker ‹addr›=‹redacted.servicebus.windows.net:9093› ‹broker›=‹0› ‹err›=dial tcp: ‹lookup cdcdrt.servicebus.windows.net on 127.0.0.53:53: no such host›
Some relevant call stacks:
// (1) ProduceSync call being blocked on waiting for the individual produce calls to finish
1 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d46c5 0x4e6348 0x4607d2f 0x48bf9a6 0x48bf8db 0x486608b 0x489e12a 0xf07f73 0xf07d06 0xf07ebe 0x489df9d 0x489dc14 0x489db3f 0x1cdf7f6 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
# 0x4d46c4 sync.runtime_Semacquire+0x24 GOROOT/src/runtime/sema.go:62
# 0x4e6347 sync.(*WaitGroup).Wait+0x47 GOROOT/src/sync/waitgroup.go:116
# 0x4607d2e github.com/twmb/franz-go/pkg/kgo.(*Client).ProduceSync+0x14e github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:263
# 0x48bf9a5 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush.func1+0x65 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:155
# 0x48bf8da github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush+0x9a github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:181
# 0x486608a github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*batchingSink).runBatchingWorker.func1+0xea github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/batching_sink.go:353
# 0x489e129 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1.1+0x129 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:265
# 0xf07f72 github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1+0x12 github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:200
# 0xf07d05 github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do+0x245 github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:177
# 0xf07ebd github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts+0xdd github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:199
# 0x489df9c github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1+0x1dc github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:260
# 0x489dc13 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2+0xb3 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:277
# 0x489db3e github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6+0x1e github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
# 0x1cdf7f5 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
// (59) stuck in condvar wait awaken
59 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x4e4872 0x4e2cf7 0x4609149 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
# 0x4d4784 sync.runtime_SemacquireMutex+0x24 GOROOT/src/runtime/sema.go:77
# 0x4e4a3c sync.(*Mutex).lockSlow+0x15c GOROOT/src/sync/mutex.go:171
# 0x4e4871 sync.(*Mutex).Lock+0x31 GOROOT/src/sync/mutex.go:90
# 0x4e2cf6 sync.(*Cond).Wait+0x96 GOROOT/src/sync/cond.go:71
# 0x4609148 github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func2+0xc8 github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:475
// (43) stuck in <-wait
43 @ 0x4a014e 0x468ddf 0x4689f2 0x4608f31 0x4608b54 0x4607d0e 0x4607cef 0x48bf9a6 0x48bf8db 0x486608b 0x489e12a 0xf07f73 0xf07d06 0xf07ebe 0x489df9d 0x489dc14 0x489db3f 0x1cdf7f6 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
# 0x4608f30 github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func3+0x110 github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:487
# 0x4608b53 github.com/twmb/franz-go/pkg/kgo.(*Client).produce+0xb73 github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:500
# 0x4607d0d github.com/twmb/franz-go/pkg/kgo.(*Client).Produce+0x12d github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:379
# 0x4607cee github.com/twmb/franz-go/pkg/kgo.(*Client).ProduceSync+0x10e github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:261
# 0x48bf9a5 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush.func1+0x65 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:155
# 0x48bf8da github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush+0x9a github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:181
# 0x486608a github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*batchingSink).runBatchingWorker.func1+0xea github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/batching_sink.go:353
# 0x489e129 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1.1+0x129 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:265
# 0xf07f72 github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1+0x12 github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:200
# 0xf07d05 github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do+0x245 github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:177
# 0xf07ebd github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts+0xdd github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:199
# 0x489df9c github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1+0x1dc github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:260
# 0x489dc13 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2+0xb3 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:277
# 0x489db3e github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6+0x1e github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
# 0x1cdf7f5 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
// (30) stuck in mu.Lock() in drainbuffered?
30 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x4608eb7 0x4608e85 0x4608b54 0x4607d0e 0x4607cef 0x48bf9a6 0x48bf8db 0x486608b 0x489e12a 0xf07f73 0xf07d06 0xf07ebe 0x489df9d 0x489dc14 0x489db3f 0x1cdf7f6 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
# 0x4d4784 sync.runtime_SemacquireMutex+0x24 GOROOT/src/runtime/sema.go:77
# 0x4e4a3c sync.(*Mutex).lockSlow+0x15c GOROOT/src/sync/mutex.go:171
# 0x4608eb6 sync.(*Mutex).Lock+0x96 GOROOT/src/sync/mutex.go:90
# 0x4608e84 github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func3+0x64 github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:483
# 0x4608b53 github.com/twmb/franz-go/pkg/kgo.(*Client).produce+0xb73 github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:500
# 0x4607d0d github.com/twmb/franz-go/pkg/kgo.(*Client).Produce+0x12d github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:379
# 0x4607cee github.com/twmb/franz-go/pkg/kgo.(*Client).ProduceSync+0x10e github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:261
# 0x48bf9a5 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush.func1+0x65 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:155
# 0x48bf8da github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*kafkaSinkClientV2).Flush+0x9a github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink_kafka_v2.go:181
# 0x486608a github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*batchingSink).runBatchingWorker.func1+0xea github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/batching_sink.go:353
# 0x489e129 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1.1+0x129 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:265
# 0xf07f72 github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1+0x12 github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:200
# 0xf07d05 github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do+0x245 github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:177
# 0xf07ebd github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts+0xdd github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:199
# 0x489df9c github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1+0x1dc github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:260
# 0x489dc13 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2+0xb3 github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:277
# 0x489db3e github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6+0x1e github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
# 0x1cdf7f5 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
// (13) stuck in mu.Lock() in the goroutine?
13 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x460911a 0x46090ff 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
# 0x4d4784 sync.runtime_SemacquireMutex+0x24 GOROOT/src/runtime/sema.go:77
# 0x4e4a3c sync.(*Mutex).lockSlow+0x15c GOROOT/src/sync/mutex.go:171
# 0x4609119 sync.(*Mutex).Lock+0x99 GOROOT/src/sync/mutex.go:90
# 0x46090fe github.com/twmb/franz-go/pkg/kgo.(*Client).produce.func2+0x7e github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:472
// (1) waiting on lock in finishing promises? not sure
1 @ 0x4a014e 0x4b30e5 0x4b30b4 0x4d4785 0x4e4a3d 0x46098fd 0x46098ca 0x460971f 0x4d8d21
# labels: {"job":"CHANGEFEED id=1004661810832080897", "n":"5"}
# 0x4d4784 sync.runtime_SemacquireMutex+0x24 GOROOT/src/runtime/sema.go:77
# 0x4e4a3c sync.(*Mutex).lockSlow+0x15c GOROOT/src/sync/mutex.go:171
# 0x46098fc sync.(*Mutex).Lock+0x13c GOROOT/src/sync/mutex.go:90
# 0x46098c9 github.com/twmb/franz-go/pkg/kgo.(*Client).finishRecordPromise+0x109 github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:587
# 0x460971e github.com/twmb/franz-go/pkg/kgo.(*producer).finishPromises+0x2be github.com/twmb/franz-go/pkg/kgo/external/com_github_twmb_franz_go/pkg/kgo/producer.go:549
I’ll keep looking at this but your assistance would be greatly appreciated
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- kgo: fix potential deadlock when reaching max buffered (records|bytes) Problem: * Record A exceeds max, is on path to block * Record B finishes concurrently * Record A's context cancels * Record A's ... — committed to twmb/franz-go by twmb 2 years ago
- kgo: fix potential deadlock when reaching max buffered (records|bytes) Problem: * Record A exceeds max, is on path to block * Record B finishes concurrently * Record A's context cancels * Record A's ... — committed to grafana/franz-go by twmb 2 years ago
- cherry-pick latest upstream changes (#10) * Fix typo in kgo.Client.ResumeFetchTopics() docs Signed-off-by: Mihai Todor <todormihai@gmail.com> * add `NewOffsetFromRecord` helper function * Fi... — committed to grafana/franz-go by ortuman 2 years ago
ETA today. I’ll reopen and then close one I push.
Merging my PR, but await the tag.
Standard case,
overMaxRecs || overMaxBytesbecomes false due to a record finishing.