Uploaded image for project: 'Talend DI components'
  1. Talend DI components
  2. TDI-41791

S3 to S3 perfomance is poor

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    Description

      19399 records (tweets) were read from S3 and sent back to a different S3 location with no transformation taking place. The 19399 records were distributed across 208 files. The distribution was not equal. The files ranged in size between 6.2KB to 264.4KB. The total amount of data being moved was 23MB. It took the Pipeline 22 minutes and 35 seconds to do this. The rate at which records were being processed was not consistent and there appeared to be big pauses between bursts of records.

      Attachments

        Issue Links

          is duplicated by

          Bug - TDI-42103 FileIO runs indefinitely when processing large CSV files with Text enclosure character

          • Major - There are major errors in a system function. However, the error situation is not critical. The customer's business may be impaired by it.
          • Rejected
          is parent of

          Work Item - Work item for the scum process TDI-41792 reading folder with many files provides an inconsistent feedback

          • Minor - Less important parts of the product are affected by the problem or a solution is available for avoiding major consequences. A minor impairment of the customer's business is possible.
          • In Progress

          Work Item - Work item for the scum process TDI-41793 Pipelines consuming Kafka records only seem to start if the flow of records into Kafka is relatively slow

          • Major - There are major errors in a system function. However, the error situation is not critical. The customer's business may be impaired by it.
          • New

          Activity

            People

              pteyssier pierre teyssier
              rhall Richard Hall
              qiyan liu, Wei Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: