Uploaded image for project: 'Talend Data Inventory'
  1. Talend Data Inventory
  2. TDC-1510

CustomType in Union of an AvroFile can induce a failure in dataset samples

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • All
    • [Tech debt]
    • Small

    Description

      Into an Avro Schema, if a Union contains multiple reference to a CustomType, there is a high probability of a fail during the conversion process of the Schema to a Json.

      org.apache.avro.SchemaParseException: Can't redefine: custom_enum_map.CustomType
        at org.talend.dataset.services.sample.AvroUtils$.toJson(AvroUtils.scala:74)
	at org.talend.dataset.services.sample.SemanticTypeAnalyzer.org$talend$dataset$services$sample$SemanticTypeAnalyzer$$setSchemaMetadataProps(SemanticTypeAnalyzer.scala:244)

      Flattening is failling with AVRO/OSA/MultiLevelComplexWithoutRefInUnion.avro on the tdc-sde bucket

      The hierarchy view is working fine

      Attachments

        1. error_enum.txt
          19 kB
        2. image-2019-04-01-17-08-22-974.png
          image-2019-04-01-17-08-22-974.png
          63 kB
        3. MultiLevelComplex.avro
          41 kB
        4. OneLevelComplex.avro
          10 kB
        5. onelevelenum-quality-error.png
          onelevelenum-quality-error.png
          40 kB
        6. screenshot-1.png
          screenshot-1.png
          97 kB
        7. Screenshot 2020-10-23 at 15.06.30.png
          Screenshot 2020-10-23 at 15.06.30.png
          180 kB
        8. simple_enum.avro
          0.2 kB

        Issue Links

          is cloned by

          Bug - TFD-7373 CustomType in Union of an AvroFile can induce a failure in dataset samples

          • Minor - Less important parts of the product are affected by the problem or a solution is available for avoiding major consequences. A minor impairment of the customer's business is possible.
          • Rejected

          Activity

            People

              wruchaud Wesley Ruchaud
              wruchaud Wesley Ruchaud
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: