kafkajs: v1.16.0 - Lossing data when one node of the cluster is lost
Describe the bug I open this issue to clarify if what I see is correct and must be solve by the user in some way, or is a bug. The specific configuration that i have tested:
- 3 nodes of Kafka v3.0 based on
confluentinc/cp-kafkadocker images. All of them with this configuration:
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_NUM_PARTITIONS: 3
KAFKA_OFFSETS_COMMIT_REQUIRED_ACKS: -1
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
- A single producer that try to write 60 records every 5 seconds with
sendmethod. The configuration of this producer is:- Client:
{ clientId: 'creator', brokers: ['localhost:9092', 'localhost:9093', 'localhost:9093'], enforceRequestTimeout: true, requestTimeout: 3000, connectionTimeout: 3000, }- Producer:
{ maxInFlightRequests: 1, transactionalId: 'my-transactional-producer', idempotent: true, transactionTimeout: 3000, retry: { retries: Number.MAX_SAFE_INTEGER }, }
I log every job (the result of a send process) and in the normal operation I can see something like:
The size of the test is: 60
{"level":"ERROR","timestamp":"2022-02-28T09:28:59.133Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 6)","broker":"localhost:9092","clientId":"creator","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":2,"size":123}
[
{
"topicName": "myTopic",
"partition": 0,
"errorCode": 0,
"baseOffset": "0",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 1,
"errorCode": 0,
"baseOffset": "0",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 2,
"errorCode": 0,
"baseOffset": "0",
"logAppendTime": "-1",
"logStartOffset": "0"
}
]
OPERATION 1 - 60/600
dbsave: 1.090s
The size of the test is: 60
[
{
"topicName": "myTopic",
"partition": 0,
"errorCode": 0,
"baseOffset": "20",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 1,
"errorCode": 0,
"baseOffset": "20",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 2,
"errorCode": 0,
"baseOffset": "20",
"logAppendTime": "-1",
"logStartOffset": "0"
}
]
OPERATION 2 - 120/600
dbsave: 25.286ms
In the middle of the test I stop one of the servers (not the controller of the cluster) with docker stop 85fsa..., and in the most of cases, not always, this is the log that i receive:
The size of the test is: 60
{"level":"ERROR","timestamp":"2022-02-28T09:29:09.132Z","logger":"kafkajs","message":"[Producer] Failed to send messages: Closed connection","retryCount":0,"retryTime":243}
[
{
"topicName": "myTopic",
"partition": 0,
"errorCode": 0,
"baseOffset": "40",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 2,
"errorCode": 0,
"baseOffset": "40",
"logAppendTime": "-1",
"logStartOffset": "0"
}
]
OPERATION 3 - 180/600
dbsave: 1.262s
The size of the test is: 60
[
{
"topicName": "myTopic",
"partition": 0,
"errorCode": 0,
"baseOffset": "60",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 1,
"errorCode": 0,
"baseOffset": "40",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 2,
"errorCode": 0,
"baseOffset": "60",
"logAppendTime": "-1",
"logStartOffset": "0"
}
]
OPERATION 4 - 240/600
dbsave: 9.988ms
As you can see the send to one of the servers failed, but the send method resolved with the response of only 2 server, and the send its not retried.
In the last job process you can see that the final result is that we have lost 20 messages without warning from the library:
The size of the test is: 60
[
{
"topicName": "myTopic",
"partition": 0,
"errorCode": 0,
"baseOffset": "180",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 1,
"errorCode": 0,
"baseOffset": "160",
"logAppendTime": "-1",
"logStartOffset": "0"
},
{
"topicName": "myTopic",
"partition": 2,
"errorCode": 0,
"baseOffset": "180",
"logAppendTime": "-1",
"logStartOffset": "0"
}
]
OPERATION 10 - 600/600
dbsave: 7.462ms
The snipped of code that I use to reproduce this behaviour is:
import { Kafka, Message } from 'kafkajs';
import fs from 'fs';
var access = fs.createWriteStream('test.log');
//@ts-ignore ignore is a test
process.stdout.write = process.stderr.write = access.write.bind(access);
function createMessages(from: number, to: number): Message[] {
const messages: Message[] = [];
for (let index = from; index < to; index += 1) {
messages.push({ value: JSON.stringify({ index }) });
}
return messages;
}
const client = new Kafka({
clientId: 'creator',
brokers: ['localhost:9092', 'localhost:9093', 'localhost:9093'],
enforceRequestTimeout: true,
requestTimeout: 3000,
connectionTimeout: 3000,
});
const producer = client.producer({
maxInFlightRequests: 1,
transactionalId: 'my-transactional-producer',
idempotent: true,
transactionTimeout: 3000,
retry: { retries: Number.MAX_SAFE_INTEGER },
});
const consumer = client.consumer({ groupId: 'myGroup' });
(async () => {
await producer.connect();
await consumer.connect();
})();
let pending = false;
let consuming = false;
let count = 0;
let aggregatedValue = 0;
let messagesCount = 0;
const messageBatchSize = 60;
const messageFinalAmount = 60 * 10;
const expectedResult = (messageFinalAmount * (messageFinalAmount - 1)) / 2;
console.log(`The final result should be ${expectedResult}`);
setInterval(async () => {
if (count === messageFinalAmount / messageBatchSize && !consuming) {
await consumer.subscribe({ topic: 'myTopic', fromBeginning: true });
await consumer.run({
eachMessage: async ({ message }) => {
if (!message.value) {
throw new Error(`TEST IS BROKEN`);
}
const value = JSON.parse(message.value.toString()).index;
aggregatedValue = aggregatedValue + value;
console.log(
`${value} - ${aggregatedValue} - ${
aggregatedValue === expectedResult ? 'ALL MESSAGE HAS BEEN RECEIVED' : 'PENDING ...'
}`
);
},
});
consuming = true;
return;
}
if (pending || consuming) {
return;
}
const messages = createMessages(messagesCount, messagesCount + messageBatchSize);
console.log(`The size of the test is: ${messages.length}`);
console.time('dbsave');
pending = true;
try {
const result = await producer.send({ topic: 'myTopic', acks: -1, messages, timeout: 3000 });
console.log(JSON.stringify(result, null, 2));
count++;
messagesCount += messageBatchSize;
console.log(`OPERATION ${count} - ${messagesCount}/${messageFinalAmount}`);
console.timeEnd('dbsave');
pending = false;
} catch (error) {
console.log(`ERROR IN OPERATION ${error.message}`);
console.timeEnd('dbsave');
pending = false;
}
}, 5000);
And the docker-compose file:
version: '3.8'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:latest
ports:
- '32181:32181'
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
kafka-1:
image: confluentinc/cp-kafka:latest
ports:
- '9092:9092'
depends_on:
- zookeeper-1
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:32181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:29092,EXTERNAL://localhost:9092
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_NUM_PARTITIONS: 3
KAFKA_OFFSETS_COMMIT_REQUIRED_ACKS: -1
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
kafka-2:
image: confluentinc/cp-kafka:latest
ports:
- '9093:9093'
depends_on:
- zookeeper-1
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:32181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:29093,EXTERNAL://localhost:9093
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_NUM_PARTITIONS: 3
KAFKA_OFFSETS_COMMIT_REQUIRED_ACKS: -1
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
kafka-3:
image: confluentinc/cp-kafka:latest
ports:
- '9094:9094'
depends_on:
- zookeeper-1
environment:
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:32181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-3:29094,EXTERNAL://localhost:9094
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_NUM_PARTITIONS: 3
KAFKA_OFFSETS_COMMIT_REQUIRED_ACKS: -1
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
kowl:
image: quay.io/cloudhut/kowl:master
depends_on:
- zookeeper-1
- kafka-1
- kafka-2
- kafka-3
environment:
KAFKA_BROKERS: kafka-1:29092,kafka-2:29093, kafka-3:29094
ports:
- "8080:8080"
Expected behavior I expect the method rejects with an error in order to retry the operations by my shelve or try to find the server that its the responsible of the partition now.
Observed behavior
Even one of the servers fails in the middle of the process of sending new records to the cluster, the send method resolves without error and the data is lost.
Environment:
- OS: Linux Ubuntu 20.04
- KafkaJS version: 1.16.0 and 1.15.0
- Kafka version: 3.0
- NodeJS version: 14.18.2
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 2
- Comments: 15 (2 by maintainers)
Sorry, any movement on this? Sounds like something that might happen on any production system that uses this library
There is a all settled() implementation in /utils.
On Thu, 10 Mar 2022, 23:09 Carlos Jiménez Saiz, @.***> wrote: