Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-2107

Implement a stop strategy for streaming input connectors

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • All
    • Hide

      See linked issues for constraints, future enhancements and potential bug fixes.

      Show
      See linked issues for constraints, future enhancements and potential bug fixes.
    • GreenHopper Ranking:
      0|i2e9uz:
    • 9223372036854775807
    • Small

    Description

      Description

      Today, a mapper is marked as infinite once started, and it never stops until the job/pipeline is killed.
      In some cases, we may need to stop an infinite mapper under some conditions :

      • an amount of records read
      • a restricted activity time frame

      We should implement the conditional stop of job in the Input lifecycle if some parameter are defined.

      Acceptance Criteria

      • Following JVM parameter can be used to stop the streaming when the number of received records has reached the value:
        • -D<plugin>.talend.input.streaming.maxRecords
      • Following JVM parameter can be used to stop the streaming when the running time of the stream has reached the value:
        • -D<plugin>.talend.input.streaming.maxDurationMs
      • The streaming should be stopped as soon as one of the parameter is reached.
      • Value -1 deactivate the parameters

      Notes:

      • max duration time is expressed in milliseconds. Usually under 1000ms, time activity won't be respected due to job latency. So always, use more than 1s.
      • If the component reading process is blocked like when using a blocking queue, the time condition may not be respected. See linked feature for overriding the stuck process.

      DoR

      The feature is "Ready" when all DoR check marks are 

      Possible status of the checkmarks:

      •  ready, complete
      •  missing some information
      Topic Description DoR
      Description Is the description enough for all stakeholders?
      Acceptance Criteria Are they defined?
      Were they validated by PO, Dev and QA?
      Jira information Is the Jira information correct? (fix version...)
      Environment Environment ready ?
      Support SSL ?
      Supports Proxy (at least SOCKs) ?
      Reachable for QA/Doc/automation(TTP, Junit)
      License Is license EE or SE clearly identified ?
      Technical Analysis Does the developer understand how it will be implemented?
      Do we have a solution?
      Approved/discussed with architecture (in-team, global or security, depending on the scope)?
      Dependencies Are all dependencies linked to JIRA (link "depends on")?
      Are they done?
      Are cross-team impact identified?
      Including SRE/Devops/IT
      Migrations Is migration needed?
      Doc Is DOCT created and linked under the epic?
      Communication channel Is slack private "feat-" channel created?
      Are stakeholders in the channel ? (QA/Doc/PO/SM)
      UX Are there changes in the UI & were they added in the DOCT?
      For new forms, is wireframe available from PM?
      For new connectors, was a TUX ticket created for a new Icon?

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              emmanuel_g emmanuel gallois
              emmanuel gallois, Fabien Desiles
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: