Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-2132

Optimisation for preparation

Apply templateInsert Lucidchart DiagramXMLWordPrintable

    • Icon: Work Item Work Item
    • Resolution: Fixed
    • Icon: Major Major
    • 1.46.0
    • None
    • schema-record
    • All
    • GreenHopper Ranking:
      0|i2fouj:
    • 9223372036854775807
    • Small

      Preparation function spend lot of time with building new Record with new Schema.
      Optimisation could be made on TCK Record & Record.Builder

      Working track.
      getEntry
      Record.BuilderImpl contains a Map<String, Entry> that is only instanciated when a schema is provided and which is not used in public function (let the client to parse all entries before finding the right one).
      So, a function like "Entry getEntry(String name)" could be welcomed to help.

      withNewSchema
      Builder.withNewSchema use code like :

              newSchema.getAllEntries()
                      .filter(e -> Objects.equals(schema.getEntry(e.getName()), e))
      

      Knowing that schema.getEntry(name) function scan a list before finding the right Entry. This mean that is schema contains N fields and new Schema M, it will make NxM comparisons. (By example, if schema contains 50 fields, and new schema 51 (add one field), it will make 2550 comparison).
      Puting schema entries in a Map for example will reduce it to MxLog(N) for a treeMap and would be statictically more efficient with a HashMap.
      (this function is not used by processing connector)

      First performance test
      For 100 000 records with 100 fields each, with uppercase processor on 1 columns (adding new col)
      (average for 4 try on each case)

      Current TCK Optimized TCK
      23561 9127

      More than more than 62% time saved.

            pteyssier pierre teyssier
            clesaec Christophe LeSaec
            Christophe LeSaec, emmanuel gallois, Fabien Desiles
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: