camel-kafka-connector: Not able to run aws s3 source connector
I am trying to run the aws s3 source connector. I have followed this documetation to install connector: https://camel.apache.org/camel-kafka-connector/latest/try-it-out-on-openshift-with-strimzi.html.
I have installed the connectors correctly as I can list it connecor:
kubectl exec -i kafka-cluster-kafka-0 -n rucvan -- curl -X GET http://my-connect-cluster-connect-api:8083/connector-plugins
Defaulting container name to kafka.
Use 'kubectl describe pod/kafka-cluster-kafka-0 -n rucvan' to see all of the containers in this pod.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0[{"class":"io.confluent.connect.s3.source.S3SourceConnector","type":"source","version":"1.2.1"},{"class":"org.apache.camel.kafkaconnector.CamelSinkConnector","type":"sink","version":"0.0.1-SNAPSHOT"},{"class":"org.apache.camel.kafkaconnector.CamelSourceConnector","type":"source","version":"0.0.1-SNAPSHOT"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"2.3.0"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"sink","version":"2.3.0"}]
This is the json for source connector:
{
"name": "s3-connector-camel",
"config": {
"connector.class": "org.apache.camel.kafkaconnector.CamelSourceConnector",
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.camel.kafkaconnector.converters.S3ObjectConverter",
"camel.source.kafka.topic": "s3-topic",
"camel.source.url": "aws-s3://{BucketName}?autocloseBody=false",
"camel.component.aws-s3.configuration.access-key": "*****",
"camel.component.aws-s3.configuration.secret-key": "*****",
"camel.source.maxPollDuration": 10000
}
}
Now when I am trying to print status for the connector, I get following error:
{"name":"s3-connector-camel","connector":{"state":"RUNNING","worker_id":"100.80.22.122:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"100.80.22.122:8083","trace":"org.apache.kafka.connect.errors.ConnectException: Failed to create and start Camel context\n\tat org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:98)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:199)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.secret-key because cannot find component with name aws-s3. Make sure you have the component on the classpath\n\tat org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$14(BaseMainSupport.java:895)\n\tat org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1084)\n\tat org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:892)\n\tat org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:545)\n\tat org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:422)\n\tat org.apache.camel.main.Main.doInit(Main.java:108)\n\tat org.apache.camel.support.service.ServiceSupport.init(ServiceSupport.java:80)\n\tat org.apache.camel.support.service.ServiceSupport.start(ServiceSupport.java:108)\n\tat org.apache.camel.main.MainSupport.run(MainSupport.java:77)\n\tat org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:214)\n\t... 3 more\n"}],"type":"source"}
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 37 (18 by maintainers)
Do you want me create document around deployment on strimzi with kubereates? I have all the example files.