julie: Julie doesn't like Placement contraints
Describe the bug I am trying to use Julie together with replica placement contraints and that seems to raise some errors.
To Reproduce
My brokers have the following config:
confluent.log.placement.constraints={"version": 1,"replicas":[{"count": 1,"constraints":{"rack": "rack-1"}},{"count": 1,"constraints":{"rack": "rack-2"}},{"count": 1,"constraints":{"rack": "rack-3"}}]}
So if I create a topic test1 manually - the constraints will be respected:
$ kafka-topics --bootstrap-server broker1:9092 --create --topic testcontstraints
Created topic testcontstraints.
$ kafka-topics --bootstrap-server broker1:9092 --describe --topic testcontstraints
Topic: testcontstraints PartitionCount: 1 ReplicationFactor: 3 Configs: compression.type=snappy,min.insync.replicas=2,segment.bytes=1073741824,confluent.placement.constraints={"version":1,"replicas":[{"count":1,"constraints":{"rack":"rack-1"}},{"count":1,"constraints":{"rack":"rack-2"}},{"count":1,"constraints":{"rack":"rack-3"}}],"observers":[]}
Topic: testcontstraints Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Offline:
However if I create now a topic via julie using the following descriptor:
descriptor.yaml
---
context: "julie"
source: "test"
projects:
- name: "schulung"
topics:
- name: "constraints"
Sadly the constraints seem not to be respected:
$ kafka-topics --bootstrap-server broker1:9092 --describe --topic julie.test.schulung.constraints
Topic: julie.test.schulung.constraints PartitionCount: 3 ReplicationFactor: 2 Configs: compression.type=snappy,min.insync.replicas=2,segment.bytes=1073741824
Topic: julie.test.schulung.constraints Partition: 0 Leader: 2 Replicas: 2,3 Isr: 2,3 Offline:
Topic: julie.test.schulung.constraints Partition: 1 Leader: 3 Replicas: 3,1 Isr: 3,1 Offline:
Topic: julie.test.schulung.constraints Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2 Offline:
If I go further and try to force a constraint on the topic descriptor like this:
---
context: "julie"
source: "test"
projects:
- name: "schulung"
topics:
- name: "constraints2"
config:
confluent.placement.constraints: '{"version": 1,"replicas":[{"count": 1,"constraints":{"rack": "rack-1"}},{"count": 1,"constraints":{"rack": "rack-2"}},{"count": 1,"constraints":{"rack": "rack-3"}}]}'
I get this error
julie-ops --brokers broker1:9092 --clientConfig prod.properties --topology descriptor4.yaml
[INFO ] 2021-03-29 23:31:56.867 [main] TopologyBuilderAdminClientBuilder - Connecting AdminClient to broker:9092
[INFO ] 2021-03-29 23:31:56.867 [main] TopologyBuilderAdminClientBuilder - Connecting AdminClient to broker1:9092
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[ERROR] 2021-03-29 23:31:59.632 [main] TopologyBuilderAdminClient - java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
[ERROR] 2021-03-29 23:31:59.632 [main] TopologyBuilderAdminClient - java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
[ERROR] 2021-03-29 23:31:59.634 [main] ExecutionPlan - Something happen running action {
"Action" : "create",
"Operation" : "com.purbon.kafka.topology.actions.topics.SyncTopicAction",
"Topic" : "julie.test.schulung.constraints2"
}
java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:196) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.actions.topics.SyncTopicAction.syncTopic(SyncTopicAction.java:60) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.actions.topics.SyncTopicAction.run(SyncTopicAction.java:46) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:88) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:69) [julie-ops.jar:?]
at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:189) [julie-ops.jar:?]
at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:201) [julie-ops.jar:?]
at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:196) [julie-ops.jar:?]
at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:144) [julie-ops.jar:?]
at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:134) [julie-ops.jar:?]
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[julie-ops.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[julie-ops.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ~[julie-ops.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:202) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:191) ~[julie-ops.jar:?]
... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
[ERROR] 2021-03-29 23:31:59.634 [main] ExecutionPlan - Something happen running action {
"Action" : "create",
"Operation" : "com.purbon.kafka.topology.actions.topics.SyncTopicAction",
"Topic" : "julie.test.schulung.constraints2"
}
java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:196) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.actions.topics.SyncTopicAction.syncTopic(SyncTopicAction.java:60) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.actions.topics.SyncTopicAction.run(SyncTopicAction.java:46) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:88) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:69) [julie-ops.jar:?]
at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:189) [julie-ops.jar:?]
at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:201) [julie-ops.jar:?]
at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:196) [julie-ops.jar:?]
at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:144) [julie-ops.jar:?]
at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:134) [julie-ops.jar:?]
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[julie-ops.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[julie-ops.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ~[julie-ops.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:202) ~[julie-ops.jar:?]
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:191) ~[julie-ops.jar:?]
... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
Exception in thread "main" java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:196)
at com.purbon.kafka.topology.actions.topics.SyncTopicAction.syncTopic(SyncTopicAction.java:60)
at com.purbon.kafka.topology.actions.topics.SyncTopicAction.run(SyncTopicAction.java:46)
at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:88)
at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:69)
at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:189)
at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:201)
at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:196)
at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:144)
at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:134)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:202)
at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:191)
... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (8 by maintainers)
I am actually happy to change it if useful for others! π
Missatge de Fobhep @.***> del dia dt., 30 de marΓ§ 2021 a les 11:17:
β Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/