-
New Feature
-
Resolution: Done
-
Major
-
None
-
All
-
-
GreenHopper Ranking:0|i2e9uz:
-
9223372036854775807
-
Small
Description
Today, a mapper is marked as infinite once started, and it never stops until the job/pipeline is killed.
In some cases, we may need to stop an infinite mapper under some conditions :
- an amount of records read
- a restricted activity time frame
We should implement the conditional stop of job in the Input lifecycle if some parameter are defined.
Acceptance Criteria
- Following JVM parameter can be used to stop the streaming when the number of received records has reached the value:
- -D<plugin>.talend.input.streaming.maxRecords
- Following JVM parameter can be used to stop the streaming when the running time of the stream has reached the value:
- -D<plugin>.talend.input.streaming.maxDurationMs
- The streaming should be stopped as soon as one of the parameter is reached.
- Value -1 deactivate the parameters
Notes:
- max duration time is expressed in milliseconds. Usually under 1000ms, time activity won't be respected due to job latency. So always, use more than 1s.
- If the component reading process is blocked like when using a blocking queue, the time condition may not be respected. See linked feature for overriding the stuck process.
DoR
The feature is "Ready" when all DoR check marks are
Possible status of the checkmarks:
- ready, complete
- missing some information
Topic | Description | DoR |
Description | Is the description enough for all stakeholders? | |
Acceptance Criteria | Are they defined? Were they validated by PO, Dev and QA? |
|
Jira information | Is the Jira information correct? (fix version...) | |
Environment | Environment ready ? Support SSL ? Supports Proxy (at least SOCKs) ? Reachable for QA/Doc/automation(TTP, Junit) |
|
License | Is license EE or SE clearly identified ? | |
Technical Analysis | Does the developer understand how it will be implemented? Do we have a solution? Approved/discussed with architecture (in-team, global or security, depending on the scope)? |
|
Dependencies | Are all dependencies linked to JIRA (link "depends on")? Are they done? Are cross-team impact identified? Including SRE/Devops/IT |
|
Migrations | Is migration needed? | |
Doc | Is DOCT created and linked under the epic? | |
Communication channel | Is slack private "feat-" channel created? Are stakeholders in the channel ? (QA/Doc/PO/SM) |
|
UX | Are there changes in the UI & were they added in the DOCT? For new forms, is wireframe available from PM? For new connectors, was a TUX ticket created for a new Icon? |
- is related to
-
TCOMP-2276 Make Streaming stop strategy configurable
- Done
-
TCOMP-2226 Implement a default UI for streaming sources for user configuration of a StopStrategy
- Done
-
TCOMP-2234 Override blocking read process in streaming connectors
- Done
-
TCOMP-2291 Provide to streaming connectors the stop strategy applied
- Done
-
TCOMP-2236 Streaming stop strategy in livy jobs
- Done
-
TCOMP-2237 Create a streaming configuration section for documentation
- Done
-
TCOMP-2276 Make Streaming stop strategy configurable
- Done