Talend DI
  1. Talend DI
  2. TDI-17967

EBCDICInput to Database - wrong column names

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.4, 5.0.2
    • Component/s: None
    • Labels:
      None
    • Found in Release:
    • Steps to reproduce:
      Hide

      To recreate:
      1) Take the SimpleOccurs.cpy and create a new copybook in the metadata tree
      2) right-click the created copybook and select retrieve schema
      3) in the schema view export the schema to XML

      Show
      To recreate: 1) Take the SimpleOccurs.cpy and create a new copybook in the metadata tree 2) right-click the created copybook and select retrieve schema 3) in the schema view export the schema to XML
    • Product Build:
      4.2.3
    • OTRS ID:
      2011100410000158
    • SLA Level:
      Gold
    • Zone Support:
      EMEA
    • Platform:
      All

      Description

      There is an issue when you try to insert the data from an EBCDIC file to a database:

      • the schema of the EBCDIC file contains an occurs-part and is stored in the repository
      • the target table does not yet exists and you want the job to create the table

      In this case the database output component (tried with DB2 an Oracle) resolves the database column names from the input rows wrong.

      "occurs" in a copybook means that the following part is repeated n times. See attached cpy-files for example: they are syntactically different but semantically equals.

      The describes misbehavior juts happens, when you use the copybook with the occurs.

      I just exported the schema of the SimpleOccurs copybook to XML. There you will see, that the "originalDbColumnName" is set wrong. So it must be an issue with the schema wizard for copybooks.

      Please see attached job + copybook + screen-shot as example.

      1. NoOccurs.cpy
        0.4 kB
        Corinne Chaumont
      2. SimpleOccurs_Schema.xml
        2 kB
        Corinne Chaumont
      3. SimpleOccurs.cpy
        0.2 kB
        Corinne Chaumont

        Activity

        Hide
        Corinne Chaumont added a comment -

        from customer to justify the priority change:
        "hence we have to process very large ebcdic files with 1000+ columns the impact is pretty high. A manual correction of that much column names takes a lot of time.

        I think to fix this issue isn't that complicated: the field "originalDbColumnName"just shouldn't be filled anyway since a copybook is a file and not a database table (for a delimited file this field is empty too)."

        Show
        Corinne Chaumont added a comment - from customer to justify the priority change: "hence we have to process very large ebcdic files with 1000+ columns the impact is pretty high. A manual correction of that much column names takes a lot of time. I think to fix this issue isn't that complicated: the field "originalDbColumnName"just shouldn't be filled anyway since a copybook is a file and not a database table (for a delimited file this field is empty too)."
        Hide
        Corinne Chaumont added a comment -

        Please consider this bug for resolution as it is important for the customer

        Show
        Corinne Chaumont added a comment - Please consider this bug for resolution as it is important for the customer
        Hide
        Nicolas Rousseau added a comment -

        I don't think this really needs a patch since we can edit manually the db columns, there is nothing blocking here.

        Show
        Nicolas Rousseau added a comment - I don't think this really needs a patch since we can edit manually the db columns, there is nothing blocking here.
        Hide
        Nicolas Rousseau added a comment - - edited

        for dev:

        DB column names are now generated with original names from the copy book file.
        Which generates:

        COLUMN01
        COLUMN02
        COLUMN03
        COLUMN04
        COLUMN03
        COLUMN04
        COLUMN03
        COLUMN04

        best would be to add an index for any duplicate:

        COLUMN01
        COLUMN02
        COLUMN03
        COLUMN04
        COLUMN03_1
        COLUMN04_1
        COLUMN03_2
        COLUMN04_2

        Show
        Nicolas Rousseau added a comment - - edited for dev: DB column names are now generated with original names from the copy book file. Which generates: COLUMN01 COLUMN02 COLUMN03 COLUMN04 COLUMN03 COLUMN04 COLUMN03 COLUMN04 best would be to add an index for any duplicate: COLUMN01 COLUMN02 COLUMN03 COLUMN04 COLUMN03_1 COLUMN04_1 COLUMN03_2 COLUMN04_2
        Hide
        yunjie gao added a comment -

        Test on Talend-All-r71296-V5.0.0NB:it is ok.

        Show
        yunjie gao added a comment - Test on Talend-All-r71296-V5.0.0NB:it is ok.
        Hide
        yunjie gao added a comment -

        Test on Talend-All-r71296-V4.2.4NB:it is ok.

        Show
        yunjie gao added a comment - Test on Talend-All-r71296-V4.2.4NB:it is ok.
        Hide
        Corinne Chaumont added a comment -

        Customer applied the patch and has the following remark:
        "I have just one question which is in combination with the Bug TDI-17966 /support ticket 2011100410000131 (Copybook retrieve schema of occurs-columns). Why do use a different indexing mechanism for the "originalDbColumnName" compared to the internal column column indexing?

        In my opinion the "originalDbColumnName" just shouldn't be filled for a copybook. For a delimited file it isn't filled either. What do you think?"

        Show
        Corinne Chaumont added a comment - Customer applied the patch and has the following remark: "I have just one question which is in combination with the Bug TDI-17966 /support ticket 2011100410000131 (Copybook retrieve schema of occurs-columns). Why do use a different indexing mechanism for the "originalDbColumnName" compared to the internal column column indexing? In my opinion the "originalDbColumnName" just shouldn't be filled for a copybook. For a delimited file it isn't filled either. What do you think?"
        Hide
        Nicolas Rousseau added a comment -

        the "original" is only here as information yes, it's not too usefull.

        There is no indexing for the original, it's just the name of the "original" column from the copybook without indexing...

        Show
        Nicolas Rousseau added a comment - the "original" is only here as information yes, it's not too usefull. There is no indexing for the original, it's just the name of the "original" column from the copybook without indexing...
        Hide
        Nicolas Rousseau added a comment - - edited

        for dev > as said from support, need another modification for this to remove the value in the original db column name

        Show
        Nicolas Rousseau added a comment - - edited for dev > as said from support, need another modification for this to remove the value in the original db column name

          People

          • Assignee:
            Nicolas Rousseau
            Reporter:
            Corinne Chaumont
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: