Details
-
Work Item
-
Status: closed
-
Minor
-
Resolution: Done
-
None
-
All
-
DGA Sprint 9 (25/6 to 12/7)
-
GreenHopper Ranking:0|i20cyz:
-
9223372036854775807
-
Small
-
2
Description
As a Data Preparation user, in Runtime Convergence mode,
I want to apply some simple functions
and run my preparation on a remote engine with a dataset as destination
In order to benefit from dataset common features (e.g. output connectivity, Quality, Trust Score, reuse it as input of a preparation or a pipeline)
Objective: Integrate migrated TCK functions in Data Preparation (development supported by DQ China) - Backend part
Why?
The Data Prep functions are migrated one by one to TCK (connectors-ee).
The adaptations have been implemented in Runtime Convergence track 2 to display in the playground only the functions available on the RE, using TCK Proxy.
When using a custom RE with the new migrated functions, the new migrated functions should appear in Data Preparation UI, but adaptations are needed in order to:
- Apply the function (and see it added to the preparation recipe)
- Run the preparation on the RE with this function
Scope: Batch 1 - simple functions
List of functions:
- ChangeNumberFormat
- Split
- Coalesce
- Normalize
- DurationConverter
- ComputeLength
(Full detail in follow-up sheet)
How?
- Implement converters in TDP PreparationV2 service for each migrated function
- Implement mapping TCK -> TDP in order to add the function (see
TDP-9836) - Implement mapping TDP -> TCK in order to build the pipeline to run the preparation on the RE using the migrated TCK function (see
TDP-9927) - access to be granted to preparation common repository for DQ China (group dq_cn)
- Implement mapping TCK -> TDP in order to add the function (see
Inputs
- Function migration follow-up sheet: https://talend.app.box.com/file/820071687915 => new status to follow the integration: "Validation" -> "Ready to integrate" -> "Integration in progress" -> "Migrated"
Acceptance criteria
- Support and reviews for DQ China team activity - integration of TCK functions into Data Preparation (TDQ-19539)
- Full QA validation will be performed separately following the Engineering (integration tests & manual)
Target acceptance criteria (indicative)
In Runtime Convergence mode, for each function in the scope:
- Function is visible in Data Prep playground
- Function can be applied to the preparation (add function) -> step created and visible in the recipe
- Preparation can be run and output is conform to the expected behavior of the function
Scenario: Run a preparation with migrated function(s)
Given a tenant with Runtime Convergence activated and a RE with migrated functions,
a dataset dataset1 (ex: S3 or local connection)
and a dataset dataset2 created with name DATASET_OUTPUT_FULLRUN (ex: S3 or local connection)
and a preparation based on dataset1
When user applies the migrated function(s)
And presses the "Export" button
Then the preparation result is exported to dataset2 DATASET_OUTPUT_FULLRUN with the correct output (ex: check the sample)
Attachments
Issue Links
- has to be done after
-
TDP-9927 [RunConv][Backend] Run a preparation on RE using a simple TCK function
-
- closed
-