Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-2127 Fix avro records where array contains nullable array
  3. TCOMP-2128

Check Avro file with Array containing null value fails with ComponentException: Not an array

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • Backlog Task
    • Resolution: Fixed
    • Minor
    • None
    • None
    • None
    • None
    • Small

    Description

      Test to reproduce

      Observation

      Pipeline fails with following exception:

      Caused by: org.talend.sdk.component.api.exception.ComponentException: (org.apache.avro.AvroRuntimeException) Not an array: [{"type":"array","items":[{"type":"array","items":{"type":"record","name":"talend_12124944014","namespace":"records","doc":"","fields":[{"name":"f1","type":["string","null"],"doc":"f1"}]}},"null"]},"null"]
          at org.apache.avro.Schema.getElementType(Schema.java:289) ~[beam-sdks-java-harness-2.34.0.jar:?]
          at org.talend.components.common.stream.input.avro.AvroToRecord.buildArrayField(AvroToRecord.java:118) ~[?:?]
          at org.talend.components.common.stream.input.avro.AvroToRecord.avroToRecord(AvroToRecord.java:81) ~[?:?]
          at org.talend.components.common.stream.input.avro.AvroToRecord.toRecord(AvroToRecord.java:64) ~[?:?]
          at org.talend.components.common.collections.IteratorMap.next(IteratorMap.java:64) ~[?:?]
          at org.talend.components.common.collections.CloseableIterator.next(CloseableIterator.java:44) ~[?:?]
          at org.talend.components.common.collections.IteratorFlatmap.next(IteratorFlatmap.java:48) ~[?:?]
          at org.talend.components.google.storage.input.GoogleStorageSource.next(GoogleStorageSource.java:69) ~[?:?]
          at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
          at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
          at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
          at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
          at org.talend.sdk.component.runtime.base.LifecycleImpl.doInvoke(LifecycleImpl.java:83) ~[component-runtime-impl-1.42.0.jar:?]
          at org.talend.sdk.component.runtime.input.InputImpl.readNext(InputImpl.java:82) ~[component-runtime-impl-1.42.0.jar:?]
          at org.talend.sdk.component.runtime.input.InputImpl.next(InputImpl.java:64) ~[component-runtime-impl-1.42.0.jar:?]
          at org.talend.datastreams.beam.compiler.schema.TPDTalendIO$BoundedReaderImpl.advance(TPDTalendIO.java:361) ~[data-processing-runtime-streamsjob-2.14.1.jar:2.14.1]
          at org.talend.datastreams.beam.compiler.schema.TPDTalendIO$BoundedReaderImpl.start(TPDTalendIO.java:356) ~[data-processing-runtime-streamsjob-2.14.1.jar:2.14.1]
          at org.apache.beam.runners.spark.io.SourceRDD$Bounded$ReaderToIteratorAdapter.seekNext(SourceRDD.java:217) ~[beam-runners-spark-3-2.34.0.jar:?]
          at org.apache.beam.runners.spark.io.SourceRDD$Bounded$ReaderToIteratorAdapter.tryProduceNext(SourceRDD.java:190) ~[beam-runners-spark-3-2.34.0.jar:?]
          ... 71 more

      Attachments

        Activity

          People

            fdesiles Fabien Desiles
            fdesiles Fabien Desiles
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: