tiflow: Compare with v6.6.0, v7.0.0 has 140%-180% performance regression with errors on sysbench oltp_write_only workload

What did you do?

deploy 2 TiDB and cdc
run sysbench oltp_write_only

What did you expect to see?

No performance regression, the cdc_changefeed_checkpoint_lag is similar.

What did you see instead?

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)
v7.0.0

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)
v7.0.0

TiCDC version (execute cdc version):

(paste TiCDC version here)
v7.0.0

About this issue

Original URL
State: closed
Created a year ago
Comments: 15 (11 by maintainers)

Commits related to this issue

sink(ticdc): add max allowed packet parmameters in sink dsn (#8627) close pingcap/tiflow#8622 — committed to pingcap/tiflow by amyangfei a year ago
sink(ticdc): add max allowed packet parmameters in sink dsn (#8627) (#8632) close pingcap/tiflow#8622 — committed to pingcap/tiflow by ti-chi-bot a year ago
sink(ticdc): turn off multi statement sql by default (#8652) close pingcap/tiflow#8622 — committed to pingcap/tiflow by amyangfei a year ago
sink(ticdc): add max allowed packet parmameters in sink dsn (#8627) close pingcap/tiflow#8622 — committed to 3AceShowHand/tiflow by amyangfei a year ago

Most upvoted comments

We met many packet errors in dmesg log

[Wed Mar 22 21:40:41 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:40:41 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:41:13 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:41:13 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:41:13 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:41:13 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:45:00 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:45:00 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:45:00 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt
[Wed Mar 22 21:45:00 2023] hns3 0000:bd:00.0 enp189s0f0: L3/L4 error pkt

The message enp189s0f0: L3/L4 error pkt in dmesg -T indicates that there was a Layer 3/Layer 4 (L3/L4) error packet encountered on network interface enp189s0f0.

Layer 3 refers to the network layer in the OSI model, which is responsible for routing and forwarding of data packets between different networks. Layer 4 refers to the transport layer in the OSI model, which provides end-to-end communication services and ensures the reliable delivery of data between hosts.

The error message suggests that there was an issue with the packet’s routing or delivery at the network or transport layer. This could be due to various reasons such as incorrect configuration, network congestion, faulty hardware, or software issues.

We should investigate the cause of the error further to determine if there is a problem with the network infrastructure or the configuration.

amyangfei on Mar 22, 2023