Uploaded image for project: 'Talend Data Prep'
  1. Talend Data Prep
  2. TDP-9977

[RunConv][Backend] Integrate migrated TCK functions - simple [batch 1]

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • 2

    Description

      As a Data Preparation user, in Runtime Convergence mode,
      I want to apply some simple functions
      and run my preparation on a remote engine with a dataset as destination
      In order to benefit from dataset common features (e.g. output connectivity, Quality, Trust Score, reuse it as input of a preparation or a pipeline)

      Objective: Integrate migrated TCK functions in Data Preparation (development supported by DQ China) - Backend part

      Why?

      The Data Prep functions are migrated one by one to TCK (connectors-ee).
      The adaptations have been implemented in Runtime Convergence track 2 to display in the playground only the functions available on the RE, using TCK Proxy.
      When using a custom RE with the new migrated functions, the new migrated functions should appear in Data Preparation UI, but adaptations are needed in order to:

      • Apply the function (and see it added to the preparation recipe)
      • Run the preparation on the RE with this function

      Scope: Batch 1 - simple functions
      List of functions:

      • ChangeNumberFormat
      • Split
      • Coalesce
      • Normalize
      • DurationConverter
      • ComputeLength

      (Full detail in follow-up sheet)

      How?

      • Implement converters in TDP PreparationV2 service for each migrated function
        • Implement mapping TCK -> TDP in order to add the function (see TDP-9836)
        • Implement mapping TDP -> TCK in order to build the pipeline to run the preparation on the RE using the migrated TCK function (see TDP-9927)
        • access to be granted to preparation common repository for DQ China (group dq_cn)

      Inputs

      Acceptance criteria

      • Support and reviews for DQ China team activity - integration of TCK functions into Data Preparation (TDQ-19539)
      • Full QA validation will be performed separately following the Engineering (integration tests & manual)
      Target acceptance criteria (indicative)

      In Runtime Convergence mode, for each function in the scope:

      • Function is visible in Data Prep playground
      • Function can be applied to the preparation (add function) -> step created and visible in the recipe
      • Preparation can be run and output is conform to the expected behavior of the function

      Scenario: Run a preparation with migrated function(s)
      Given a tenant with Runtime Convergence activated and a RE with migrated functions,
      a dataset dataset1 (ex: S3 or local connection)
      and a dataset dataset2 created with name DATASET_OUTPUT_FULLRUN (ex: S3 or local connection)
      and a preparation based on dataset1
      When user applies the migrated function(s)
      And presses the "Export" button
      Then the preparation result is exported to dataset2 DATASET_OUTPUT_FULLRUN with the correct output (ex: check the sample)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              odubois Olivier Dubois
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: