Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-2701

Endless runing job with lookup join

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • None
    • None
    • None
    • All
    • Small

    Description

      From pgautier :

      I was checking performance test for the Join component (connectors-ee) -> TDP-11987.
      To validate that we are still able to use a lookup dataset with 100K rows and 10 columns.
      It seems that it does not work anymore with the engine I used (2024-03).
      I dig a bit more to understand and here is my analysis :

      • we are not able to use a lookup dataset with 10K rows (it works only for really small dataset).
        The job never ends without any information in livy container.
      • I configured a RE with an old configuration (TCK version 1.62) and it works with big files.
      • I suspect this PR to be the root cause related to this ticket TCOMP-2339
        This was supposed to fix the multi-thread execution for the Join component but it seems to brake join with medium/big dataset

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              emmanuel_g emmanuel gallois
              Axel Catoire Axel Catoire
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: