Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-235

Provide a way for components to perform some processes on driver node only, in distributed systems.

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • New Feature
    • Resolution: Duplicate
    • Major
    • 0.18.0
    • None
    • None
    • None
    • All
    • Hide

      It duplicates TCOMP-234

      Show
      It duplicates TCOMP-234
    • GreenHopper Ranking:
      0|i0vpir:
    • 9223372036854775807
    • Small
    • 3

    Description

      New components framework is designed to provide possibility to run tasks in distributed environments. Components Runtimes might be run on different nodes in a cluster. But sometimes we have some processes that should be performed on driver nodes only, for example some processes which are dependent on file system(some files should be uploaded/downloaded to/from file system).
      We have to provide some method/interface to run such processes on driver node and not to delegate run of those tasks to cluster nodes.

      (IG): UPD
      RuntimableRuntime interface was derived from SourceOrSink. Its initialize() method is supposed to be called on driver node.
      So, there is an API method for some piece of work to be done on Driver node (only preprocessing ).
      Also we may need a method for postprocessing stuff, but it is not clear for the moment.

      As for full job, which should be run on driver node. We don't know examples for the moment. We don't know if it is required for us.
      Also there is List<? extends BoundedSource> splitIntoBundles() method. Component may return one Source in the returned List to tell that job should be performed on single node.

      So, may we consider this task as finished (RuntimableRuntime interface was created?)
      Also there is ComponentDriverInitialization interface

      Attachments

        Issue Links

          Activity

            People

              dchmyga Dmytro Chmyga (Inactive)
              dchmyga Dmytro Chmyga (Inactive)
              Dmytro Chmyga (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: