datahub: oracle etl job fetch a error

backend service logs

[info] p.c.s.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:19000
2018-05-03 18:34:20 INFO  application:55 - Creating Pool for datasource 'wherehows'                              
2018-05-03 18:34:21 INFO  p.a.d.DefaultDBApi:45 - Database [wherehows] connected at jdbc:mysql://192.168.241.17:3306/wherehows
2018-05-03 18:34:22 INFO  application:34 - on start---===                                                        
2018-05-03 18:34:22 INFO  p.a.Play:97 - Application started (Dev)                                                
2018-05-03 18:34:22 INFO  application:58 - Kafka jobs: []
2018-05-03 18:34:22 INFO  application:61 - Enabled jobs: [ORACLE_METADATA_ETL]
2018-05-03 18:34:22 INFO  application:146 - Terminating KafkaConsumerMaster...
2018-05-03 18:34:22 INFO  application:64 - Scheduled jobs: {}                                                    
2018-05-03 18:39:22 INFO  application:61 - Enabled jobs: [ORACLE_METADATA_ETL]                                   
2018-05-03 18:39:22 INFO  application:64 - Scheduled jobs: {ORACLE_METADATA_ETL=1525343700}
2018-05-03 18:39:22 INFO  application:82 - Running job: ORACLE_METADATA_ETL                                      
2018-05-03 18:39:22 INFO  application:91 - Current running jobs : 
2018-05-03 18:39:22 INFO  application:98 - Send message : (jobName:ORACLE_METADATA_ETL, whEtlExecId:4)
2018-05-03 18:39:22 DEBUG application:72 - run command : [java, -cp, , -Dconfig=/root/data/wherehows/exec/4.properties, -DCONTEXT=ORACLE_METADATA_ETL, -Dlogback.configurationFile=etl_logback.xml, -DLOG_DIR=/root/data/wherehows, metadata.etl.Launcher] ; timeout: 600
2018-05-03 18:39:22 ERROR application:121 - *** Process + 22569 failed, status: 1                                
2018-05-03 18:39:22 ERROR application:122 - Error Details:                                                       

java.lang.Exception: Process + 22569 failed
        at actors.EtlJobActor.onReceive(EtlJobActor.java:123)
        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
        at akka.actor.ActorCell.invoke(ActorCell.scala:487)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
        at akka.dispatch.Mailbox.run(Mailbox.scala:220)
2018-05-03 18:39:22 ERROR application:138 - ETL job (jobName:ORACLE_METADATA_ETL, whEtlExecId:4) got a problem
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

ORACLE_METADATA_ETL.job

# Common ETL configs                                                                                             
job.class=metadata.etl.dataset.oracle.OracleMetadataEtl                                                          
job.cron.expr=0 0/5 * * * ? *                                                                                    
job.timeout=600                                                                                                  
#job.cmd.params=                                                                                                 
#job.disabled=1                                                                                                  
job.ref.id=1892                                                                                                  
                                                                                                                 
# Oracle JDBC driver                                                                                             
oracle.db.driver=oracle.jdbc.OracleDriver                                                                        
                                                                                                                 
# Oracle DB name                                                                                                 
oracle.db.name=HR                                                                                                
                                                                                                                 
# Oracle username                                                                                                
oracle.db.username=system                                                                                        
                                                                                                                 
# Oracle password                                                                                                
oracle.db.password=oracle                                                                                        
                                                                                                                 
# Oracle JDBC URL                                                                                                
oracle.db.jdbc.url=jdbc:oracle:thin:@192.168.241.21:1521:xe                                                      
                                                                                                                 
# Place to store datasets metadata csv file                                                                      
oracle.metadata=oracle.csv                                                                                       
                                                                                                                 
# Place to store dataset fields csv file                                                                         
oracle.field_metadata=oracle_field.csv                                                                           
                                                                                                                 
# Place to store the sample data csv file                                                                        
oracle.sample_data=oracle_sample.csv                                                                             
                                                                                                                 
# Whether to get sample data (True/False)                                                                        
oracle.load_sample=False                                                                                         
                                                                                                                 
# List of excluded databases in oracle                                                                           
oracle.exclude_db=                                                                                               
                                                                                                                 
# innodb_lock_wait_timeout when accessing MySQL Db
innodb_lock_wait_timeout=1500

wherehows/ORACLE_METADATA_ETL.stderr

Error: Could not find or load main class metadata.etl.Launcher

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19

Most upvoted comments

@xkcoding I have created a detailed document as to how you can use the docker option end to end. I have it tested. Let me know if you need it.

@ankurgadgilwar Oh, Genius! I so appreciate what you’ve done. The steps are very detailed. But I use Apache Atlas to store metadata and use JDBC way to collect metadata. After I finished this project, I will try your solution by using wherehows. Thanks again!

@ankurgadgilwar if I find a resolution, I will update this issue. 😄