Talend DI

Intermittent ConcurrentModificationException in runtime stats package

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Incomplete
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: ??? unknown ???
  • Labels:
    None
  • Environment:

    Seen on Windows 7 Pro 32 bit Java 1.6
    Seen on Solaris 10 Java 1.6

  • Found in Release:
  • Steps to reproduce:
    Hide

    Unknown - this is intermittent.
    The jobs in question used tMap, with lookups that store-temp-data to disk.
    The problem appears to occur more during memory shortages, particularly when the garbage collector is running frequently.

    Show
    Unknown - this is intermittent. The jobs in question used tMap, with lookups that store-temp-data to disk. The problem appears to occur more during memory shortages, particularly when the garbage collector is running frequently.
  • Product Build:
    63143
  • Java version:
    1.6
  • Platform:
    All
  • Operating System:
    Several
  • Resolution comment:
    Hide

    Incomplete issue. Feel free to open it again if needed, and if you can provide a way to reproduce.

    Thanks.

    Show
    Incomplete issue. Feel free to open it again if needed, and if you can provide a way to reproduce. Thanks.

Description

The following exception has occurred several times today on my laptop and a customer's testing machine.
The jobs in question are characterised by tMaps with multiple lookups that store temp data on disk, and also in that they have often been garbage collecting a lot and running out of memory. (We have been stressing the job in order to tune memory consumption).

The error is intermittent. I believe that it was suppressed by disabling stats collection for the job but since it's intermittent I cannot be 100% certain of this.

Stack trace is:

Starting job tMap_lookup_on_disk at 22:00 18/08/2011.

[statistics] connecting to socket on port 3896
[statistics] connected

        • job runs without issues for nearly 1 minute, processes many rows ****
        • ... and then it failed as follows ****

Exception in component
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
at java.util.LinkedList$ListItr.next(Unknown Source)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.stopThreadStat(RunStat.java:225)
at caci.tmap_lookup_on_disk_0_1.tMap_lookup_on_disk.tFileInputDelimited_4Process(tMap_lookup_on_disk.java:3115)
at caci.tmap_lookup_on_disk_0_1.tMap_lookup_on_disk.tFileInputDelimited_2Process(tMap_lookup_on_disk.java:1361)
at caci.tmap_lookup_on_disk_0_1.tMap_lookup_on_disk.runJobInTOS(tMap_lookup_on_disk.java:3287)
at caci.tmap_lookup_on_disk_0_1.tMap_lookup_on_disk.main(tMap_lookup_on_disk.java:3161)
[statistics] disconnected
Job tMap_lookup_on_disk ended at 22:01 18/08/2011. [exit code=1]

Activity

Hide
Rémy DUBOIS added a comment -

Can you please attach your job. In the same time, can you please test the job again, de-activating the statistics ?

Thanks.

Show
Rémy DUBOIS added a comment - Can you please attach your job. In the same time, can you please test the job again, de-activating the statistics ? Thanks.
Hide
Imad Rahman added a comment -

I can't provide exports of these but I will try to reproduce the issue in a job of my own. If I succeed then I will post the testcase.

We ran the jobs again with deactivated statistics, and they worked fine. But bear in mind that the problem was intermittent so this doesn't 100% prove the problem is the stats - I originally thought it was a problem in the stats because of the stack trace.

Show
Imad Rahman added a comment - I can't provide exports of these but I will try to reproduce the issue in a job of my own. If I succeed then I will post the testcase. We ran the jobs again with deactivated statistics, and they worked fine. But bear in mind that the problem was intermittent so this doesn't 100% prove the problem is the stats - I originally thought it was a problem in the stats because of the stack trace.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: