Uploaded image for project: 'Talend Data Prep'
  1. Talend Data Prep
  2. TDP-4691

[Performance][On Premise][Windows] Service unavailable for lookup, fetchMore and Fullrun

Apply templateInsert Lucidchart Diagram
    XMLWordPrintable

Details

    • All
    • Sprint 21 TDP, Sprint 22 TDP, Sprint 23 TDP (feb 26), Sprint 24 TDP (mar 19)
    • Hide

      Has been fixed by other fixes closing java streams and also many fixes on TDP performance.

      Show
      Has been fixed by other fixes closing java streams and also many fixes on TDP performance.
    • Small

    Description

      When running performance on master version with 10 users making different actions during around 1 hour, I got many errors 500 in middle/end of the run.

      Looking into logs I've found errors about Service unavailable on 3 kind of actions : lookup, fetchMore and Fullrun.
      Here are the extracts of the logs for the lookup :

      2017-11-20 10:01:18.560 ERROR [user a4178003-1b51-4da9-bc62-511c228f4717] 2612 --- [managedTaskEngine-1] o.t.d.t.a.c.CompileDataSetRowAction      : Unable to use action 'lookup' due to unexpected error.
      ...
      org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSetLookupRowMatcher': Invocation of init method failed; nested exception is org.talend.dataprep.exception.TDPException: Unable to retrieve dataset content
      	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
      ...
      Caused by: org.talend.dataprep.exception.TDPException: Unable to retrieve dataset content
      	at org.talend.dataprep.command.dataset.DataSetGet.lambda$new$116(DataSetGet.java:65) ~[dataprep-backend-service-2.3.0-SNAPSHOT.jar:na]
      ...
      Caused by: org.talend.dataprep.exception.TDPException: Service Unavailable
      	at org.talend.dataprep.exception.ExceptionsConfiguration$ExceptionsConversions.lambda$doWith$137(ExceptionsConfiguration.java:86) ~[dataprep-backend-service-2.3.0-SNAPSHOT.jar:na]
      ...
      

      For a fullrun :

      2017-11-20 09:42:38.792  INFO [user a4178003-1b51-4da9-bc62-511c228f4717] 2612 --- [managedTaskEngine-5] t.d.f.e.PreparationFullRunExportStrategy : Asynchronous export for ExportParameters{exportType='CSV', preparationId='8cfd039b-9fcf-4a4b-b7b7-c5815d3406b8', stepId='$lastStepId', datasetId='912248fc-dcad-4e1e-947d-ee3eaa092821', from='null', exportName='Test_user7@dataprep.com-37 PREPARATION', arguments={exportParameters.fileName=Test_user7@dataprep.com-37 PREPARATION, exportParameters.csvSeparator=;}, filter=null} ended.
      2017-11-20 09:42:38.796 ERROR [user a4178003-1b51-4da9-bc62-511c228f4717] 2612 --- [managedTaskEngine-5] o.t.d.async.SimpleManagedTaskExecutor    : Execution 8cfd039b-9fcf-4a4b-b7b7-c5815d3406b8-b40dd27b-264f-4fc5-becb-852880de7cd9 finished with error.
      ...
      org.talend.dataprep.exception.TDPException: Sorry an unexpected error occurred and we could not complete your last operation, but you can keep using Data Preparation
      ...
      Caused by: org.talend.dataprep.exception.TDPException: Unable to retrieve dataset content
      ...
      Caused by: org.talend.dataprep.exception.TDPException: Service Unavailable
      ...
      

      For a fetch more :

      2017-11-20 09:37:19.454  INFO [user 61478569-7d2c-4934-9fbb-8e5ec39ce977] 2612 --- [managedTaskEngine-1] o.t.d.t.s.sample.FilterSamplingStrategy  : Sampling for preparation
      ...
      2017-11-20 09:37:19.457 ERROR [user 61478569-7d2c-4934-9fbb-8e5ec39ce977] 2612 --- [managedTaskEngine-1] o.t.d.async.SimpleManagedTaskExecutor    : Execution 61478569-7d2c-4934-9fbb-8e5ec39ce977-ed7340f7-74a8-435f-af7c-0f1cbef6000d-61f48a4d-f5de-4719-9238-e6b8d200cf8e finished with error.
      ...
      org.talend.dataprep.exception.TDPException: Sorry an unexpected error occurred and we could not complete your last operation, but you can keep using Data Preparation
      	at org.talend.dataprep.transformation.service.sample.FilterSamplingStrategy.sample(FilterSamplingStrategy.java:117) ~[dataprep-ee-transformation-2.3.0-SNAPSHOT-lib.jar:na]
      ...
      Caused by: org.talend.dataprep.exception.TDPException: Unable to retrieve dataset content
      	at org.talend.dataprep.command.dataset.DataSetGet.lambda$new$116(DataSetGet.java:65) ~[dataprep-backend-service-2.3.0-SNAPSHOT.jar:na]
      ...
      Caused by: org.talend.dataprep.exception.TDPException: Service Unavailable
      	at org.talend.dataprep.exception.ExceptionsConfiguration$ExceptionsConversions.lambda$doWith$137(ExceptionsConfiguration.java:86) ~[dataprep-backend-service-2.3.0-SNAPSHOT.jar:na]
      

      Don't know if it's linked but I also had 59 errors of type
      Here is the log of the error and it seems to happen after dataset creation, preparation creation, data sampling, async export... a little bit everywhere in the logs in fact :

      2017-11-20 09:42:53.142  WARN [user ] 2612 --- [SimpleAsyncTaskExecutor-382] o.t.dataprep.security.RunAsTaskExecutor  : Error when running 
      
      java.lang.IllegalArgumentException: Cannot dispatch without an AsyncContext
      	at org.springframework.util.Assert.notNull(Assert.java:134) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE]
      	at org.springframework.web.context.request.async.StandardServletAsyncWebRequest.dispatch(StandardServletAsyncWebRequest.java:122) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
      	at org.springframework.web.context.request.async.WebAsyncManager.setConcurrentResultAndDispatch(WebAsyncManager.java:353) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
      	at org.springframework.web.context.request.async.WebAsyncManager.access$200(WebAsyncManager.java:58) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
      	at org.springframework.web.context.request.async.WebAsyncManager$4.run(WebAsyncManager.java:324) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
      	at org.talend.dataprep.security.RunAsTaskExecutor.lambda$toRunnable$4(RunAsTaskExecutor.java:139) ~[dataprep-ee-backend-service-2.3.0-SNAPSHOT.jar:na]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.8.0_141]
      	at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_141]
      	at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_141]
      

      Attachments

        1. 20180316_mongo01.zip
          5.73 MB
        2. 20180316_mongo02.zip
          5.25 MB
        3. 20180316_perf_tests.zip
          1.55 MB
        4. 20180320_perf_tests.zip
          1.51 MB
        5. 20180321_perf_tests.zip
          1.56 MB
        6. 20180912_perf_tests.zip
          1.41 MB
        7. app.log.tar.gz
          252 kB
        8. app - Copy.zip
          350 kB
        9. concurrency-1516030006942.zip
          540 kB
        10. concurrencywithoutdq-1536831414765LINUX.zip
          1.12 MB
        11. gatling (1).log.tar.gz
          539 kB
        12. gatling1.log
          7.52 MB
        13. gatling2.log
          7.16 MB
        14. screenshot-1.png
          screenshot-1.png
          74 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              vferreira Vincent Ferreira
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: