Uploaded image for project: 'Talend Data Prep'
  1. Talend Data Prep
  2. TDP-7824

[on Prem 7.3.1] Export streams failed

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • Bug
    • Status: Rejected
    • Major
    • Resolution: Won't Do
    • None
    • None
    • None

    Description

      On VM TDP on prem 7.3.1 : SUR-RND-009, export full run using streams/SJS fail :

      Version :

      • - components-api-service-rest-all-components-0.27.3
      • - data-streams-ee-streamsrunner-svc-0.6.3.2
      • - spark-jobserver-0.6.6.4

      import HDFS are OK

      Full log TDP attached.

      The errors are in e47df011-b85a-4980-ac16-50f30f425b24\spark-job-server.log/

      Below the error :

       

      [2019-10-23 08:53:55,597] [ERROR] s.j.JobManagerActor [] [akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431] - Got Throwable[2019-10-23 08:53:55,597] [ERROR] s.j.JobManagerActor [] [akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431] - Got Throwablejava.util.ServiceConfigurationError: org.talend.components.api.ComponentInstaller: Provider org.talend.components.dataprep.DataPrepFamilyDefinition could not be instantiated at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:581) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:803) at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:721) at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1394) at org.talend.components.service.spi.ServiceSpiFactory.getDefinitionRegistry(ServiceSpiFactory.java:57) at org.talend.datastreams.beam.compiler.catalog.impl.InstalledComponentService.getDefinitionRegistry(InstalledComponentService.java:22) at org.talend.datastreams.beam.compiler.catalog.impl.StaticDefinitionCatalog.getDefinition(StaticDefinitionCatalog.java:43) at org.talend.datastreams.beam.compiler.catalog.impl.StaticDefinitionCatalog.getDatasetDefinition(StaticDefinitionCatalog.java:39) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.getDatasetDefinition(ComponentsUtil.java:161) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.getDatasetProperties(ComponentsUtil.java:301) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.lambda$buildReferences$3(ComponentsUtil.java:194) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.buildReferences(ComponentsUtil.java:194) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.<init>(ComponentsUtil.java:55) at org.talend.datastreams.beam.compiler.runtimeflow.RuntimeFlowBeamCompiler.compile(RuntimeFlowBeamCompiler.java:67) at org.talend.datastreams.beam.compiler.runtimeflow.RuntimeFlowBeamCompiler.compile(RuntimeFlowBeamCompiler.java:29) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:83) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:22) at spark.jobserver.SparkJobBase$class.runJob(SparkJob.scala:31) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:22) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:22) at spark.jobserver.JobManagerActor$$anonfun$getJobFuture$4.apply(JobManagerActor.scala:310) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)Caused by: java.lang.ExceptionInInitializerError at org.talend.components.api.component.ComponentDefinition.<clinit>(ComponentDefinition.java:73) at org.talend.components.api.component.AbstractComponentDefinition.<init>(AbstractComponentDefinition.java:68) at org.talend.components.dataprep.DataPrepDefinition.<init>(DataPrepDefinition.java:25) at org.talend.components.dataprep.tdatasetinput.TDataSetInputDefinition.<init>(TDataSetInputDefinition.java:36) at org.talend.components.dataprep.DataPrepFamilyDefinition.<init>(DataPrepFamilyDefinition.java:37) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:779) ... 25 moreCaused by: java.lang.NullPointerException at org.apache.commons.lang3.SystemUtils.isJavaVersionAtLeast(SystemUtils.java:1413) at org.apache.commons.lang3.ClassUtils.isAssignable(ClassUtils.java:594) at org.apache.commons.lang3.reflect.TypeUtils.isAssignable(TypeUtils.java:376) at org.apache.commons.lang3.reflect.TypeUtils.getTypeArguments(TypeUtils.java:900) at org.apache.commons.lang3.reflect.TypeUtils.getTypeArguments(TypeUtils.java:798) at org.apache.commons.lang3.reflect.TypeUtils.getTypeArguments(TypeUtils.java:784) at org.apache.commons.lang3.reflect.TypeLiteral.<init>(TypeLiteral.java:91) at org.talend.daikon.properties.property.PropertyFactory$1.<init>(PropertyFactory.java:29) at org.talend.daikon.properties.property.PropertyFactory.<clinit>(PropertyFactory.java:29) ... 35 more[2019-10-23 08:53:55,606] [ERROR] s.j.JobManagerActor [] [akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431] - Exception from job e47df011-b85a-4980-ac16-50f30f425b24: java.util.concurrent.ExecutionException: Boxed Error at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244) at scala.concurrent.Promise$class.complete(Promise.scala:55) at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)Caused by: java.util.ServiceConfigurationError: org.talend.components.api.ComponentInstaller: Provider org.talend.components.dataprep.DataPrepFamilyDefinition could not be instantiated at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:581) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:803) at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:721) at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1394) at org.talend.components.service.spi.ServiceSpiFactory.getDefinitionRegistry(ServiceSpiFactory.java:57) at org.talend.datastreams.beam.compiler.catalog.impl.InstalledComponentService.getDefinitionRegistry(InstalledComponentService.java:22) at org.talend.datastreams.beam.compiler.catalog.impl.StaticDefinitionCatalog.getDefinition(StaticDefinitionCatalog.java:43) at org.talend.datastreams.beam.compiler.catalog.impl.StaticDefinitionCatalog.getDatasetDefinition(StaticDefinitionCatalog.java:39) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.getDatasetDefinition(ComponentsUtil.java:161) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.getDatasetProperties(ComponentsUtil.java:301) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.lambda$buildReferences$3(ComponentsUtil.java:194) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.buildReferences(ComponentsUtil.java:194) at org.talend.datastreams.beam.compiler.util.ComponentsUtil.<init>(ComponentsUtil.java:55) at org.talend.datastreams.beam.compiler.runtimeflow.RuntimeFlowBeamCompiler.compile(RuntimeFlowBeamCompiler.java:67) at org.talend.datastreams.beam.compiler.runtimeflow.RuntimeFlowBeamCompiler.compile(RuntimeFlowBeamCompiler.java:29) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:83) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:22) at spark.jobserver.SparkJobBase$class.runJob(SparkJob.scala:31) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:22) at org.talend.datastreams.sjs.jobs.DatastreamsJob$.runJob(DatastreamsJob.scala:22) at spark.jobserver.JobManagerActor$$anonfun$getJobFuture$4.apply(JobManagerActor.scala:310) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ... 3 moreCaused by: java.lang.ExceptionInInitializerError at org.talend.components.api.component.ComponentDefinition.<clinit>(ComponentDefinition.java:73) at org.talend.components.api.component.AbstractComponentDefinition.<init>(AbstractComponentDefinition.java:68) at org.talend.components.dataprep.DataPrepDefinition.<init>(DataPrepDefinition.java:25) at org.talend.components.dataprep.tdatasetinput.TDataSetInputDefinition.<init>(TDataSetInputDefinition.java:36) at org.talend.components.dataprep.DataPrepFamilyDefinition.<init>(DataPrepFamilyDefinition.java:37) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:779) ... 25 moreCaused by: java.lang.NullPointerException at org.apache.commons.lang3.SystemUtils.isJavaVersionAtLeast(SystemUtils.java:1413) at org.apache.commons.lang3.ClassUtils.isAssignable(ClassUtils.java:594) at org.apache.commons.lang3.reflect.TypeUtils.isAssignable(TypeUtils.java:376) at org.apache.commons.lang3.reflect.TypeUtils.getTypeArguments(TypeUtils.java:900) at org.apache.commons.lang3.reflect.TypeUtils.getTypeArguments(TypeUtils.java:798) at org.apache.commons.lang3.reflect.TypeUtils.getTypeArguments(TypeUtils.java:784) at org.apache.commons.lang3.reflect.TypeLiteral.<init>(TypeLiteral.java:91) at org.talend.daikon.properties.property.PropertyFactory$1.<init>(PropertyFactory.java:29) at org.talend.daikon.properties.property.PropertyFactory.<clinit>(PropertyFactory.java:29) ... 35 more[2019-10-23 08:53:55,607] [INFO ] s.j.JobStatusActor [] [akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431/$a] - Job e47df011-b85a-4980-ac16-50f30f425b24 finished with an error[2019-10-23 08:53:55,613] [INFO ] s.j.JobManagerActor [] [] - Shutting down SparkContext 0cd01dfb-org.talend.datastreams.sjs.jobs.DatastreamsJob[2019-10-23 08:53:55,613] [INFO ] a.a.LocalActorRef [] [akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431/$a] - Message [spark.jobserver.CommonMessages$Unsubscribe] from Actor[akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431#-1885012508] to Actor[akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431/$a#797529965] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.[2019-10-23 08:53:55,617] [ERROR] o.a.s.s.LiveListenerBus [] [akka://JobServer/user/jobManager-8c-ac8e-d369d44ea431] - Listener SparkEndOfRunMetricsListener threw an exceptionjava.lang.UnsupportedOperationException: empty.reduceLeft at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:178) at scala.collection.AbstractTraversable.reduceLeft(Traversable.scala:104) at scala.collection.TraversableOnce$class.reduce(TraversableOnce.scala:206) at scala.collection.AbstractTraversable.reduce(Traversable.scala:104) at org.talend.datastreams.sjs.utils.SparkEndOfRunMetricsListener.totalAggregation(JobMetrics.scala:36) at org.talend.datastreams.sjs.utils.SparkEndOfRunMetricsListener.onApplicationEnd(JobMetrics.scala:25) at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:54) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1181) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
       
      

       

      Attachments

        1. bug HDFS sur-rnd-009.7z
          60 kB
          Arnaud Secher
        2. image-2019-10-23-09-38-39-108.png
          17 kB
          Arnaud Secher

        Issue Links

          Activity

            People

              Unassigned Unassigned
              asecher Arnaud Secher
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: