Endless runing job with lookup join

I was checking performance test for the Join component (connectors-ee) -> TDP-11987.
To validate that we are still able to use a lookup dataset with 100K rows and 10 columns.
It seems that it does not work anymore with the engine I used (2024-03).
I dig a bit more to understand and here is my analysis :

• we are not able to use a lookup dataset with 10K rows (it works only for really small dataset).
  The job never ends without any information in livy container.
• I configured a RE with an old configuration (TCK version 1.62) and it works with big files.
• I suspect this PR to be the root cause related to this ticket TCOMP-2339
  This was supposed to fix the multi-thread execution for the Join component but it seems to brake join with medium/big dataset