Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-896

s3 and hdfs connector throw "java.lang.OutOfMemoryError: Requested array size exceeds VM limit" exception if fetch all data from data set

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • 0.24.0
    • None
    • None
    • None
    • All
    • Small

    Description

      When use tdatasetinput to fetch data set from data-catalog, it will fetch all data, and the component rest server get a null "limit" value from data-catalog(s3, hdfs)/dataprep(s3), then set Integer.MAX_VALUE to the data set runtime interface:

      @Override
          public void getSample(int limit, Consumer<IndexedRecord> consumer) {}
      

      Then the exception appear.

      Attachments

        Activity

          People

            pteyssier pierre teyssier
            wwang Wei Wang
            Wei Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: