Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-1226

Tolerate NULL fields in avro schema

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • Work Item
    • Resolution: Fixed
    • Minor
    • None
    • None
    • None
    • All
    • GreenHopper Ranking:
      0|i1dm2b:
    • 9223372036854775807
    • Small

    Description

      When used as a library AvroSchema can fail with this exception: 

      org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalArgumentException: No enum constant org.talend.sdk.component.api.record.Schema.Type.NULL
       at org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:66)
       at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:99)
       at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87)
       at org.talend.datastreams.beam.compiler.BeamJob.fullRun(BeamJob.java:269)
       at org.talend.datastreams.beam.compiler.BeamJob.fullRun(BeamJob.java:295)
       at org.talend.datastreams.streamsjob.FullRunJob$.runJob(FullRunJob.scala:89)
       at org.talend.datastreams.streamsjob.FullRunJob$.main(FullRunJob.scala:63)
       at org.talend.datastreams.streamsjob.FullRunJob.main(FullRunJob.scala)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
       at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:162)
       at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:160)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:422)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
       at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:160)
       at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
       at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
       at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
       Caused by: java.lang.IllegalArgumentException: No enum constant org.talend.sdk.component.api.record.Schema.Type.NULL
       at java.lang.Enum.valueOf(Enum.java:238)
       at org.talend.sdk.component.api.record.Schema$Type.valueOf(Schema.java:37)
       at org.talend.sdk.component.runtime.beam.spi.record.AvroSchema.doMapType(AvroSchema.java:109)
       at org.talend.sdk.component.runtime.beam.spi.record.AvroSchema.mapType(AvroSchema.java:98)
       at org.talend.sdk.component.runtime.beam.spi.record.AvroSchema.lambda$getEntries$0(AvroSchema.java:75)
       at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
       at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
       at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
       at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
       at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
       at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
       at org.talend.sdk.component.runtime.beam.spi.record.AvroSchema.getEntries(AvroSchema.java:81)
       at org.talend.sdk.component.runtime.beam.spi.record.AvroSchema.toString(AvroSchema.java:32)
       at org.talend.components.salesforce.service.SalesforceOutputService.createSObject(SalesforceOutputService.java:150)
       at org.talend.components.salesforce.service.SalesforceOutputService.doInsert(SalesforceOutputService.java:305)
       at org.talend.components.salesforce.service.SalesforceOutputService.finish(SalesforceOutputService.java:486)
       at org.talend.components.salesforce.output.SalesforceOutput.release(SalesforceOutput.java:95)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.talend.sdk.component.runtime.base.LifecycleImpl.doInvoke(LifecycleImpl.java:65)
       at org.talend.sdk.component.runtime.base.LifecycleImpl.lambda$invoke$0(LifecycleImpl.java:57)
       at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
       at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
       at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
       at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
       at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
       at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
       at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
       at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
       at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
       at org.talend.sdk.component.runtime.base.LifecycleImpl.invoke(LifecycleImpl.java:57)
       at org.talend.sdk.component.runtime.base.LifecycleImpl.stop(LifecycleImpl.java:53)
       at org.talend.sdk.component.runtime.beam.BaseProcessorFn.tearDown(BaseProcessorFn.java:127)

       IMPORTANT: this is a tolerance and not a framework feature since null does not exist in the framework design

      Attachments

        1. image.png
          180 kB
          Pengyu Zhou

        Activity

          People

            rmannibucau Romain Manni-Bucau
            pyzhou Pengyu Zhou
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: