[2023-03-27T03:15:20,888] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 179.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:20,888] [INFO ] o.a.s.s.DAGScheduler - ResultStage 179 (DStream at SparkUnboundedSource.java:174) finished in 2.168 s
[2023-03-27T03:15:20,888] [INFO ] o.a.s.s.DAGScheduler - Job 155 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:20,889] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 179: Stage finished
[2023-03-27T03:15:20,889] [INFO ] o.a.s.s.DAGScheduler - Job 155 finished: DStream at SparkUnboundedSource.java:174, took 2.170467 s
[2023-03-27T03:15:20,913] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886918710 ms
[2023-03-27T03:15:20,913] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886918710 ms
[2023-03-27T03:15:20,913] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886918710 ms
[2023-03-27T03:15:20,914] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886918710 ms.0 from job set of time 1679886918710 ms
[2023-03-27T03:15:20,914] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886918710 ms
[2023-03-27T03:15:20,915] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886918710 ms to writer queue
[2023-03-27T03:15:20,916] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886918710 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886918710'
[2023-03-27T03:15:20,924] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:20,925] [INFO ] o.a.s.s.DAGScheduler - Got job 156 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:20,925] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 180 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:20,925] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:20,925] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:20,926] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 180 (MapPartitionsRDD[1812] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:20,928] [INFO ] o.a.s.SparkContext - Created broadcast 170 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:20,928] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 180 (MapPartitionsRDD[1812] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:20,929] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 180.0 with 1 tasks resource profile 0
[2023-03-27T03:15:20,930] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 180.0 (TID 156) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 24699 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:20,930] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 180.0 (TID 156)
[2023-03-27T03:15:20,933] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886893685.bk
[2023-03-27T03:15:20,934] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886918710 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886918710', took 22767 bytes and 18 ms
[2023-03-27T03:15:20,937] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:20,942] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 180.0 (TID 156). 5951 bytes result sent to driver
[2023-03-27T03:15:20,943] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 180.0 (TID 156) in 14 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:20,943] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 180.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:20,943] [INFO ] o.a.s.s.DAGScheduler - ResultStage 180 (foreach at UnboundedDataset.java:79) finished in 0.017 s
[2023-03-27T03:15:20,944] [INFO ] o.a.s.s.DAGScheduler - Job 156 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:20,944] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 180: Stage finished
[2023-03-27T03:15:20,944] [INFO ] o.a.s.s.DAGScheduler - Job 156 finished: foreach at UnboundedDataset.java:79, took 0.019332 s
[2023-03-27T03:15:20,944] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886918710 ms.0 from job set of time 1679886918710 ms
[2023-03-27T03:15:20,944] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.234 s for time 1679886918710 ms (execution: 0.030 s)
[2023-03-27T03:15:20,944] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1768 from persistence list
[2023-03-27T03:15:20,945] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1767 from persistence list
[2023-03-27T03:15:20,945] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1308 from persistence list
[2023-03-27T03:15:20,946] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1307 from persistence list
[2023-03-27T03:15:20,946] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1788 from persistence list
[2023-03-27T03:15:20,947] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1787 from persistence list
[2023-03-27T03:15:20,948] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1786 from persistence list
[2023-03-27T03:15:20,948] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1785 from persistence list
[2023-03-27T03:15:20,948] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1784 from persistence list
[2023-03-27T03:15:20,949] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1783 from persistence list
[2023-03-27T03:15:20,949] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1782 from persistence list
[2023-03-27T03:15:20,950] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1781 from persistence list
[2023-03-27T03:15:20,950] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1780 from persistence list
[2023-03-27T03:15:20,950] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1779 from persistence list
[2023-03-27T03:15:20,950] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1778 from persistence list
[2023-03-27T03:15:20,951] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1777 from persistence list
[2023-03-27T03:15:20,951] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1776 from persistence list
[2023-03-27T03:15:20,951] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1775 from persistence list
[2023-03-27T03:15:20,952] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1774 from persistence list
[2023-03-27T03:15:20,952] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1773 from persistence list
[2023-03-27T03:15:20,953] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1772 from persistence list
[2023-03-27T03:15:20,953] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1771 from persistence list
[2023-03-27T03:15:20,954] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1770 from persistence list
[2023-03-27T03:15:20,954] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1769 from persistence list
[2023-03-27T03:15:20,954] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886918710 ms
[2023-03-27T03:15:20,954] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886918710 ms
[2023-03-27T03:15:20,955] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886918710 ms
[2023-03-27T03:15:20,959] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886918710 ms to writer queue
[2023-03-27T03:15:20,959] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886918710 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886918710'
[2023-03-27T03:15:20,980] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886893685
[2023-03-27T03:15:20,981] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886918710 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886918710', took 22763 bytes and 21 ms
[2023-03-27T03:15:20,981] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886918710 ms
[2023-03-27T03:15:20,981] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886918710 ms
[2023-03-27T03:15:20,981] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:20,982] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886818610: 
[2023-03-27T03:15:20,982] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886813605 ms
[2023-03-27T03:15:20,985] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:15.737Z, highWatermark=2023-03-27T03:15:20.743Z, synchronizedProcessingTime=2023-03-27T03:15:18.710Z}}
[2023-03-27T03:15:20,985] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886918710 has completed, watermarks have been updated.
[2023-03-27T03:15:23,716] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:23,727] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:23,728] [INFO ] o.a.s.s.DAGScheduler - Got job 157 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:23,728] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 181 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:23,728] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:23,728] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:23,729] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 181 (MapPartitionsRDD[1816] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:23,731] [INFO ] o.a.s.SparkContext - Created broadcast 171 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:23,731] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 181 (MapPartitionsRDD[1816] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:23,731] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 181.0 with 1 tasks resource profile 0
[2023-03-27T03:15:23,733] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 181.0 (TID 157) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 28300 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:23,733] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 181.0 (TID 157)
[2023-03-27T03:15:23,740] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:23,742] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:25,752] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2010 millis on reading.
[2023-03-27T03:15:25,992] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 181.0 (TID 157)
[rdd_1790_0]
[2023-03-27T03:15:25,992] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 181.0 (TID 157). 7890 bytes result sent to driver
[2023-03-27T03:15:25,993] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 181.0 (TID 157) in 2261 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:25,993] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 181.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:25,994] [INFO ] o.a.s.s.DAGScheduler - ResultStage 181 (DStream at SparkUnboundedSource.java:174) finished in 2.265 s
[2023-03-27T03:15:25,994] [INFO ] o.a.s.s.DAGScheduler - Job 157 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:25,994] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 181: Stage finished
[2023-03-27T03:15:25,994] [INFO ] o.a.s.s.DAGScheduler - Job 157 finished: DStream at SparkUnboundedSource.java:174, took 2.267174 s
[2023-03-27T03:15:26,018] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886923715 ms
[2023-03-27T03:15:26,018] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886923715 ms
[2023-03-27T03:15:26,018] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886923715 ms
[2023-03-27T03:15:26,018] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886923715 ms
[2023-03-27T03:15:26,018] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886923715 ms.0 from job set of time 1679886923715 ms
[2023-03-27T03:15:26,020] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886923715 ms to writer queue
[2023-03-27T03:15:26,020] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886923715 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886923715'
[2023-03-27T03:15:26,027] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:26,027] [INFO ] o.a.s.s.DAGScheduler - Got job 158 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:26,028] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 182 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:26,028] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:26,028] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:26,028] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 182 (MapPartitionsRDD[1836] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:26,031] [INFO ] o.a.s.SparkContext - Created broadcast 172 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:26,031] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 182 (MapPartitionsRDD[1836] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:26,031] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 182.0 with 1 tasks resource profile 0
[2023-03-27T03:15:26,034] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 182.0 (TID 158) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 28300 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:26,034] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 182.0 (TID 158)
[2023-03-27T03:15:26,037] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886898690.bk
[2023-03-27T03:15:26,037] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886923715 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886923715', took 22767 bytes and 17 ms
[2023-03-27T03:15:26,043] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:26,050] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 182.0 (TID 158). 5951 bytes result sent to driver
[2023-03-27T03:15:26,050] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 182.0 (TID 158) in 17 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:26,051] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 182.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:26,051] [INFO ] o.a.s.s.DAGScheduler - ResultStage 182 (foreach at UnboundedDataset.java:79) finished in 0.022 s
[2023-03-27T03:15:26,051] [INFO ] o.a.s.s.DAGScheduler - Job 158 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:26,051] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 182: Stage finished
[2023-03-27T03:15:26,051] [INFO ] o.a.s.s.DAGScheduler - Job 158 finished: foreach at UnboundedDataset.java:79, took 0.024059 s
[2023-03-27T03:15:26,051] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886923715 ms.0 from job set of time 1679886923715 ms
[2023-03-27T03:15:26,052] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.336 s for time 1679886923715 ms (execution: 0.033 s)
[2023-03-27T03:15:26,052] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1792 from persistence list
[2023-03-27T03:15:26,052] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1791 from persistence list
[2023-03-27T03:15:26,053] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1332 from persistence list
[2023-03-27T03:15:26,053] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1331 from persistence list
[2023-03-27T03:15:26,054] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1812 from persistence list
[2023-03-27T03:15:26,054] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1811 from persistence list
[2023-03-27T03:15:26,054] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1810 from persistence list
[2023-03-27T03:15:26,054] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1809 from persistence list
[2023-03-27T03:15:26,055] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1808 from persistence list
[2023-03-27T03:15:26,055] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1807 from persistence list
[2023-03-27T03:15:26,055] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1806 from persistence list
[2023-03-27T03:15:26,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1805 from persistence list
[2023-03-27T03:15:26,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1804 from persistence list
[2023-03-27T03:15:26,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1803 from persistence list
[2023-03-27T03:15:26,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1802 from persistence list
[2023-03-27T03:15:26,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1801 from persistence list
[2023-03-27T03:15:26,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1800 from persistence list
[2023-03-27T03:15:26,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1799 from persistence list
[2023-03-27T03:15:26,058] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1798 from persistence list
[2023-03-27T03:15:26,058] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1797 from persistence list
[2023-03-27T03:15:26,058] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1796 from persistence list
[2023-03-27T03:15:26,058] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1795 from persistence list
[2023-03-27T03:15:26,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1794 from persistence list
[2023-03-27T03:15:26,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1793 from persistence list
[2023-03-27T03:15:26,059] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886923715 ms
[2023-03-27T03:15:26,059] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886923715 ms
[2023-03-27T03:15:26,060] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886923715 ms
[2023-03-27T03:15:26,061] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886923715 ms to writer queue
[2023-03-27T03:15:26,061] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886923715 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886923715'
[2023-03-27T03:15:26,079] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886898690
[2023-03-27T03:15:26,080] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886923715 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886923715', took 22763 bytes and 18 ms
[2023-03-27T03:15:26,080] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886923715 ms
[2023-03-27T03:15:26,080] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886923715 ms
[2023-03-27T03:15:26,081] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:26,081] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886823615: 
[2023-03-27T03:15:26,081] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886818610 ms
[2023-03-27T03:15:26,087] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:20.743Z, highWatermark=2023-03-27T03:15:25.752Z, synchronizedProcessingTime=2023-03-27T03:15:23.715Z}}
[2023-03-27T03:15:26,087] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886923715 has completed, watermarks have been updated.
[2023-03-27T03:15:28,721] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:28,736] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:28,736] [INFO ] o.a.s.s.DAGScheduler - Got job 159 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:28,736] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 183 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:28,736] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:28,737] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:28,737] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 183 (MapPartitionsRDD[1840] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:28,739] [INFO ] o.a.s.SparkContext - Created broadcast 173 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:28,740] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 183 (MapPartitionsRDD[1840] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:28,740] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 183.0 with 1 tasks resource profile 0
[2023-03-27T03:15:28,741] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 183.0 (TID 159) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 31901 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:28,742] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 183.0 (TID 159)
[2023-03-27T03:15:28,749] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:28,751] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:30,758] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2006 millis on reading.
[2023-03-27T03:15:30,929] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 183.0 (TID 159)
[rdd_1814_0]
[2023-03-27T03:15:30,930] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 183.0 (TID 159). 7847 bytes result sent to driver
[2023-03-27T03:15:30,932] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 183.0 (TID 159) in 2192 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:30,932] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 183.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:30,932] [INFO ] o.a.s.s.DAGScheduler - ResultStage 183 (DStream at SparkUnboundedSource.java:174) finished in 2.194 s
[2023-03-27T03:15:30,932] [INFO ] o.a.s.s.DAGScheduler - Job 159 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:30,932] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 183: Stage finished
[2023-03-27T03:15:30,933] [INFO ] o.a.s.s.DAGScheduler - Job 159 finished: DStream at SparkUnboundedSource.java:174, took 2.196717 s
[2023-03-27T03:15:30,961] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886928720 ms
[2023-03-27T03:15:30,961] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886928720 ms
[2023-03-27T03:15:30,961] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886928720 ms
[2023-03-27T03:15:30,961] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886928720 ms
[2023-03-27T03:15:30,961] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886928720 ms.0 from job set of time 1679886928720 ms
[2023-03-27T03:15:30,963] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886928720 ms to writer queue
[2023-03-27T03:15:30,963] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886928720 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886928720'
[2023-03-27T03:15:30,968] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:30,969] [INFO ] o.a.s.s.DAGScheduler - Got job 160 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:30,969] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 184 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:30,969] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:30,969] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:30,970] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 184 (MapPartitionsRDD[1860] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:30,977] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886903695.bk
[2023-03-27T03:15:30,978] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886928720 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886928720', took 22767 bytes and 14 ms
[2023-03-27T03:15:30,981] [INFO ] o.a.s.SparkContext - Created broadcast 174 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:30,982] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 184 (MapPartitionsRDD[1860] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:30,982] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 184.0 with 1 tasks resource profile 0
[2023-03-27T03:15:30,983] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 184.0 (TID 160) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 31901 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:30,983] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 184.0 (TID 160)
[2023-03-27T03:15:30,993] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:31,000] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 184.0 (TID 160). 5951 bytes result sent to driver
[2023-03-27T03:15:31,001] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 184.0 (TID 160) in 19 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:31,001] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 184.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:31,001] [INFO ] o.a.s.s.DAGScheduler - ResultStage 184 (foreach at UnboundedDataset.java:79) finished in 0.031 s
[2023-03-27T03:15:31,001] [INFO ] o.a.s.s.DAGScheduler - Job 160 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:31,001] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 184: Stage finished
[2023-03-27T03:15:31,002] [INFO ] o.a.s.s.DAGScheduler - Job 160 finished: foreach at UnboundedDataset.java:79, took 0.033064 s
[2023-03-27T03:15:31,002] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886928720 ms.0 from job set of time 1679886928720 ms
[2023-03-27T03:15:31,002] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.282 s for time 1679886928720 ms (execution: 0.041 s)
[2023-03-27T03:15:31,002] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1816 from persistence list
[2023-03-27T03:15:31,003] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1815 from persistence list
[2023-03-27T03:15:31,003] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1356 from persistence list
[2023-03-27T03:15:31,004] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1355 from persistence list
[2023-03-27T03:15:31,004] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1836 from persistence list
[2023-03-27T03:15:31,004] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1835 from persistence list
[2023-03-27T03:15:31,004] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1834 from persistence list
[2023-03-27T03:15:31,005] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1833 from persistence list
[2023-03-27T03:15:31,005] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1832 from persistence list
[2023-03-27T03:15:31,005] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1831 from persistence list
[2023-03-27T03:15:31,006] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1830 from persistence list
[2023-03-27T03:15:31,006] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1829 from persistence list
[2023-03-27T03:15:31,007] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1828 from persistence list
[2023-03-27T03:15:31,007] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1827 from persistence list
[2023-03-27T03:15:31,007] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1826 from persistence list
[2023-03-27T03:15:31,007] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1825 from persistence list
[2023-03-27T03:15:31,008] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1824 from persistence list
[2023-03-27T03:15:31,008] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1823 from persistence list
[2023-03-27T03:15:31,008] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1822 from persistence list
[2023-03-27T03:15:31,009] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1821 from persistence list
[2023-03-27T03:15:31,009] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1820 from persistence list
[2023-03-27T03:15:31,009] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1819 from persistence list
[2023-03-27T03:15:31,010] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1818 from persistence list
[2023-03-27T03:15:31,010] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1817 from persistence list
[2023-03-27T03:15:31,010] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886928720 ms
[2023-03-27T03:15:31,010] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886928720 ms
[2023-03-27T03:15:31,010] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886928720 ms
[2023-03-27T03:15:31,012] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886928720 ms to writer queue
[2023-03-27T03:15:31,012] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886928720 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886928720'
[2023-03-27T03:15:31,028] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886903695
[2023-03-27T03:15:31,029] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886928720 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886928720', took 22763 bytes and 16 ms
[2023-03-27T03:15:31,029] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886928720 ms
[2023-03-27T03:15:31,029] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886928720 ms
[2023-03-27T03:15:31,029] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:31,032] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:25.752Z, highWatermark=2023-03-27T03:15:30.758Z, synchronizedProcessingTime=2023-03-27T03:15:28.720Z}}
[2023-03-27T03:15:31,032] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886928720 has completed, watermarks have been updated.
[2023-03-27T03:15:31,035] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886828620: 
[2023-03-27T03:15:31,035] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886823615 ms
[2023-03-27T03:15:33,726] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:33,738] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:33,739] [INFO ] o.a.s.s.DAGScheduler - Got job 161 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:33,739] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 185 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:33,739] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:33,739] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:33,740] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 185 (MapPartitionsRDD[1864] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:33,742] [INFO ] o.a.s.SparkContext - Created broadcast 175 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:33,743] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 185 (MapPartitionsRDD[1864] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:33,743] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 185.0 with 1 tasks resource profile 0
[2023-03-27T03:15:33,744] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 185.0 (TID 161) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 35502 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:33,745] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 185.0 (TID 161)
[2023-03-27T03:15:33,756] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:33,758] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:35,765] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2006 millis on reading.
[2023-03-27T03:15:35,942] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 185.0 (TID 161)
[rdd_1838_0]
[2023-03-27T03:15:35,943] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 185.0 (TID 161). 7847 bytes result sent to driver
[2023-03-27T03:15:35,944] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 185.0 (TID 161) in 2201 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:35,944] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 185.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:35,945] [INFO ] o.a.s.s.DAGScheduler - ResultStage 185 (DStream at SparkUnboundedSource.java:174) finished in 2.205 s
[2023-03-27T03:15:35,945] [INFO ] o.a.s.s.DAGScheduler - Job 161 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:35,945] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 185: Stage finished
[2023-03-27T03:15:35,946] [INFO ] o.a.s.s.DAGScheduler - Job 161 finished: DStream at SparkUnboundedSource.java:174, took 2.207151 s
[2023-03-27T03:15:35,969] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886933725 ms
[2023-03-27T03:15:35,970] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886933725 ms
[2023-03-27T03:15:35,970] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886933725 ms
[2023-03-27T03:15:35,970] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886933725 ms.0 from job set of time 1679886933725 ms
[2023-03-27T03:15:35,970] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886933725 ms
[2023-03-27T03:15:35,972] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886933725 ms to writer queue
[2023-03-27T03:15:35,972] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886933725 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886933725'
[2023-03-27T03:15:35,983] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:35,984] [INFO ] o.a.s.s.DAGScheduler - Got job 162 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:35,984] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 186 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:35,984] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:35,984] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:35,985] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 186 (MapPartitionsRDD[1884] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:35,988] [INFO ] o.a.s.SparkContext - Created broadcast 176 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:35,988] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 186 (MapPartitionsRDD[1884] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:35,988] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 186.0 with 1 tasks resource profile 0
[2023-03-27T03:15:35,989] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886908700.bk
[2023-03-27T03:15:35,990] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 186.0 (TID 162) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 35502 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:35,990] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 186.0 (TID 162)
[2023-03-27T03:15:35,990] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886933725 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886933725', took 22767 bytes and 18 ms
[2023-03-27T03:15:36,006] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:36,013] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 186.0 (TID 162). 5994 bytes result sent to driver
[2023-03-27T03:15:36,014] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 186.0 (TID 162) in 25 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:36,014] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 186.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:36,015] [INFO ] o.a.s.s.DAGScheduler - ResultStage 186 (foreach at UnboundedDataset.java:79) finished in 0.030 s
[2023-03-27T03:15:36,015] [INFO ] o.a.s.s.DAGScheduler - Job 162 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:36,015] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 186: Stage finished
[2023-03-27T03:15:36,015] [INFO ] o.a.s.s.DAGScheduler - Job 162 finished: foreach at UnboundedDataset.java:79, took 0.031986 s
[2023-03-27T03:15:36,016] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886933725 ms.0 from job set of time 1679886933725 ms
[2023-03-27T03:15:36,016] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.291 s for time 1679886933725 ms (execution: 0.046 s)
[2023-03-27T03:15:36,016] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1840 from persistence list
[2023-03-27T03:15:36,017] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1839 from persistence list
[2023-03-27T03:15:36,017] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1380 from persistence list
[2023-03-27T03:15:36,018] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1379 from persistence list
[2023-03-27T03:15:36,018] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1860 from persistence list
[2023-03-27T03:15:36,019] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1859 from persistence list
[2023-03-27T03:15:36,019] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1858 from persistence list
[2023-03-27T03:15:36,019] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1857 from persistence list
[2023-03-27T03:15:36,020] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1856 from persistence list
[2023-03-27T03:15:36,020] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1855 from persistence list
[2023-03-27T03:15:36,021] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1854 from persistence list
[2023-03-27T03:15:36,021] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1853 from persistence list
[2023-03-27T03:15:36,021] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1852 from persistence list
[2023-03-27T03:15:36,022] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1851 from persistence list
[2023-03-27T03:15:36,022] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1850 from persistence list
[2023-03-27T03:15:36,022] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1849 from persistence list
[2023-03-27T03:15:36,023] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1848 from persistence list
[2023-03-27T03:15:36,023] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1847 from persistence list
[2023-03-27T03:15:36,023] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1846 from persistence list
[2023-03-27T03:15:36,024] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1845 from persistence list
[2023-03-27T03:15:36,024] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1844 from persistence list
[2023-03-27T03:15:36,024] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1843 from persistence list
[2023-03-27T03:15:36,025] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1842 from persistence list
[2023-03-27T03:15:36,025] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1841 from persistence list
[2023-03-27T03:15:36,025] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886933725 ms
[2023-03-27T03:15:36,025] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886933725 ms
[2023-03-27T03:15:36,025] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886933725 ms
[2023-03-27T03:15:36,027] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886933725 ms to writer queue
[2023-03-27T03:15:36,031] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886933725 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886933725'
[2023-03-27T03:15:36,043] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886908700
[2023-03-27T03:15:36,043] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886933725 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886933725', took 22763 bytes and 12 ms
[2023-03-27T03:15:36,043] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886933725 ms
[2023-03-27T03:15:36,043] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886933725 ms
[2023-03-27T03:15:36,043] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:36,044] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886833625: 
[2023-03-27T03:15:36,044] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886828620 ms
[2023-03-27T03:15:36,048] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:30.758Z, highWatermark=2023-03-27T03:15:35.765Z, synchronizedProcessingTime=2023-03-27T03:15:33.725Z}}
[2023-03-27T03:15:36,048] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886933725 has completed, watermarks have been updated.
[2023-03-27T03:15:38,730] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:38,746] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:38,747] [INFO ] o.a.s.s.DAGScheduler - Got job 163 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:38,747] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 187 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:38,747] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:38,748] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:38,748] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 187 (MapPartitionsRDD[1888] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:38,750] [INFO ] o.a.s.SparkContext - Created broadcast 177 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:38,751] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 187 (MapPartitionsRDD[1888] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:38,751] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 187.0 with 1 tasks resource profile 0
[2023-03-27T03:15:38,753] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 187.0 (TID 163) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 39103 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:38,754] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 187.0 (TID 163)
[2023-03-27T03:15:38,761] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:38,764] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:40,779] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2015 millis on reading.
[2023-03-27T03:15:40,942] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 187.0 (TID 163)
[rdd_1862_0]
[2023-03-27T03:15:40,943] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 187.0 (TID 163). 7847 bytes result sent to driver
[2023-03-27T03:15:40,944] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 187.0 (TID 163) in 2193 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:40,945] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 187.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:40,945] [INFO ] o.a.s.s.DAGScheduler - ResultStage 187 (DStream at SparkUnboundedSource.java:174) finished in 2.196 s
[2023-03-27T03:15:40,945] [INFO ] o.a.s.s.DAGScheduler - Job 163 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:40,945] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 187: Stage finished
[2023-03-27T03:15:40,945] [INFO ] o.a.s.s.DAGScheduler - Job 163 finished: DStream at SparkUnboundedSource.java:174, took 2.198734 s
[2023-03-27T03:15:40,980] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886938730 ms
[2023-03-27T03:15:40,980] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886938730 ms
[2023-03-27T03:15:40,980] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886938730 ms
[2023-03-27T03:15:40,980] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886938730 ms.0 from job set of time 1679886938730 ms
[2023-03-27T03:15:40,980] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886938730 ms
[2023-03-27T03:15:40,982] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886938730 ms to writer queue
[2023-03-27T03:15:40,983] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886938730 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886938730'
[2023-03-27T03:15:40,994] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:40,995] [INFO ] o.a.s.s.DAGScheduler - Got job 164 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:40,995] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 188 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:40,995] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:40,995] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:40,996] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 188 (MapPartitionsRDD[1908] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:41,029] [INFO ] o.a.s.SparkContext - Created broadcast 178 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:41,029] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886913705.bk
[2023-03-27T03:15:41,029] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 188 (MapPartitionsRDD[1908] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:41,029] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 188.0 with 1 tasks resource profile 0
[2023-03-27T03:15:41,029] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886938730 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886938730', took 22767 bytes and 47 ms
[2023-03-27T03:15:41,032] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 188.0 (TID 164) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 39103 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:41,033] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 188.0 (TID 164)
[2023-03-27T03:15:41,043] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:41,052] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 188.0 (TID 164). 5994 bytes result sent to driver
[2023-03-27T03:15:41,052] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 188.0 (TID 164) in 21 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:41,052] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 188.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:41,053] [INFO ] o.a.s.s.DAGScheduler - ResultStage 188 (foreach at UnboundedDataset.java:79) finished in 0.057 s
[2023-03-27T03:15:41,053] [INFO ] o.a.s.s.DAGScheduler - Job 164 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:41,053] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 188: Stage finished
[2023-03-27T03:15:41,053] [INFO ] o.a.s.s.DAGScheduler - Job 164 finished: foreach at UnboundedDataset.java:79, took 0.058708 s
[2023-03-27T03:15:41,053] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886938730 ms.0 from job set of time 1679886938730 ms
[2023-03-27T03:15:41,054] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.323 s for time 1679886938730 ms (execution: 0.073 s)
[2023-03-27T03:15:41,054] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1864 from persistence list
[2023-03-27T03:15:41,055] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1863 from persistence list
[2023-03-27T03:15:41,055] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1404 from persistence list
[2023-03-27T03:15:41,055] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1403 from persistence list
[2023-03-27T03:15:41,055] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1884 from persistence list
[2023-03-27T03:15:41,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1883 from persistence list
[2023-03-27T03:15:41,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1882 from persistence list
[2023-03-27T03:15:41,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1881 from persistence list
[2023-03-27T03:15:41,056] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1880 from persistence list
[2023-03-27T03:15:41,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1879 from persistence list
[2023-03-27T03:15:41,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1878 from persistence list
[2023-03-27T03:15:41,057] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1877 from persistence list
[2023-03-27T03:15:41,058] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1876 from persistence list
[2023-03-27T03:15:41,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1875 from persistence list
[2023-03-27T03:15:41,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1874 from persistence list
[2023-03-27T03:15:41,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1873 from persistence list
[2023-03-27T03:15:41,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1872 from persistence list
[2023-03-27T03:15:41,059] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1871 from persistence list
[2023-03-27T03:15:41,060] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1870 from persistence list
[2023-03-27T03:15:41,060] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1869 from persistence list
[2023-03-27T03:15:41,060] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1868 from persistence list
[2023-03-27T03:15:41,060] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1867 from persistence list
[2023-03-27T03:15:41,060] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1866 from persistence list
[2023-03-27T03:15:41,061] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1865 from persistence list
[2023-03-27T03:15:41,061] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886938730 ms
[2023-03-27T03:15:41,061] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886938730 ms
[2023-03-27T03:15:41,061] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886938730 ms
[2023-03-27T03:15:41,063] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886938730 ms to writer queue
[2023-03-27T03:15:41,063] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886938730 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886938730'
[2023-03-27T03:15:41,074] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886913705
[2023-03-27T03:15:41,074] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886938730 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886938730', took 22763 bytes and 11 ms
[2023-03-27T03:15:41,074] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886938730 ms
[2023-03-27T03:15:41,074] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886938730 ms
[2023-03-27T03:15:41,074] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:41,075] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886838630: 
[2023-03-27T03:15:41,075] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886833625 ms
[2023-03-27T03:15:41,080] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:35.765Z, highWatermark=2023-03-27T03:15:40.779Z, synchronizedProcessingTime=2023-03-27T03:15:38.730Z}}
[2023-03-27T03:15:41,080] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886938730 has completed, watermarks have been updated.
[2023-03-27T03:15:43,736] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:43,736] [INFO ] o.a.s.s.d.InternalMapWithStateDStream - Marking RDD 1910 for time 1679886943735 ms for checkpointing
[2023-03-27T03:15:43,745] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:43,746] [INFO ] o.a.s.s.DAGScheduler - Got job 165 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:43,746] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 189 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:43,746] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:43,747] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:43,747] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 189 (MapPartitionsRDD[1912] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:43,750] [INFO ] o.a.s.SparkContext - Created broadcast 179 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:43,750] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 189 (MapPartitionsRDD[1912] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:43,750] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 189.0 with 1 tasks resource profile 0
[2023-03-27T03:15:43,751] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 189.0 (TID 165) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 42704 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:43,752] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 189.0 (TID 165)
[2023-03-27T03:15:43,760] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:43,762] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:45,771] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2009 millis on reading.
[2023-03-27T03:15:45,950] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 189.0 (TID 165)
[rdd_1886_0]
[2023-03-27T03:15:45,951] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 189.0 (TID 165). 7847 bytes result sent to driver
[2023-03-27T03:15:45,952] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 189.0 (TID 165) in 2202 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:45,952] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 189.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:45,953] [INFO ] o.a.s.s.DAGScheduler - ResultStage 189 (DStream at SparkUnboundedSource.java:174) finished in 2.204 s
[2023-03-27T03:15:45,953] [INFO ] o.a.s.s.DAGScheduler - Job 165 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:45,953] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 189: Stage finished
[2023-03-27T03:15:45,953] [INFO ] o.a.s.s.DAGScheduler - Job 165 finished: DStream at SparkUnboundedSource.java:174, took 2.207418 s
[2023-03-27T03:15:45,973] [INFO ] o.a.s.SparkContext - Created broadcast 180 from DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:45,974] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:45,975] [INFO ] o.a.s.s.DAGScheduler - Got job 166 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:45,975] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 190 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:45,975] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:45,975] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:45,976] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 190 (MapWithStateRDD[1910] at mapWithState at SparkUnboundedSource.java:106), which has no missing parents
[2023-03-27T03:15:45,978] [INFO ] o.a.s.SparkContext - Created broadcast 181 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:45,979] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 190 (MapWithStateRDD[1910] at mapWithState at SparkUnboundedSource.java:106) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:45,979] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 190.0 with 1 tasks resource profile 0
[2023-03-27T03:15:45,981] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 190.0 (TID 166) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 42704 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:45,981] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 190.0 (TID 166)
[2023-03-27T03:15:45,989] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:46,005] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 190.0 (TID 166). 923 bytes result sent to driver
[2023-03-27T03:15:46,006] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 190.0 (TID 166) in 27 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:46,006] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 190.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:46,006] [INFO ] o.a.s.s.DAGScheduler - ResultStage 190 (DStream at SparkUnboundedSource.java:174) finished in 0.030 s
[2023-03-27T03:15:46,006] [INFO ] o.a.s.s.DAGScheduler - Job 166 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:46,006] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 190: Stage finished
[2023-03-27T03:15:46,006] [INFO ] o.a.s.s.DAGScheduler - Job 166 finished: DStream at SparkUnboundedSource.java:174, took 0.031984 s
[2023-03-27T03:15:46,020] [INFO ] o.a.s.r.ReliableCheckpointRDD - Checkpointing took 66 ms.
[2023-03-27T03:15:46,034] [INFO ] o.a.s.SparkContext - Created broadcast 182 from DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:46,035] [INFO ] o.a.s.r.ReliableRDDCheckpointData - Done checkpointing RDD 1910 to file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/6a51bdf3-a8f0-4ea3-b694-153664e45b0a/rdd-1910, new parent is RDD 1913
[2023-03-27T03:15:46,067] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886943735 ms
[2023-03-27T03:15:46,067] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886943735 ms
[2023-03-27T03:15:46,068] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886943735 ms
[2023-03-27T03:15:46,068] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886943735 ms
[2023-03-27T03:15:46,068] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886943735 ms.0 from job set of time 1679886943735 ms
[2023-03-27T03:15:46,070] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886943735 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886943735'
[2023-03-27T03:15:46,071] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886943735 ms to writer queue
[2023-03-27T03:15:46,075] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:46,076] [INFO ] o.a.s.s.DAGScheduler - Got job 167 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:46,076] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 191 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:46,076] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:46,076] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:46,077] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 191 (MapPartitionsRDD[1933] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:46,080] [INFO ] o.a.s.SparkContext - Created broadcast 183 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:46,080] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 191 (MapPartitionsRDD[1933] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:46,080] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 191.0 with 1 tasks resource profile 0
[2023-03-27T03:15:46,082] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 191.0 (TID 167) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 42704 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:46,082] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 191.0 (TID 167)
[2023-03-27T03:15:46,090] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886918710.bk
[2023-03-27T03:15:46,090] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886943735 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886943735', took 22740 bytes and 19 ms
[2023-03-27T03:15:46,091] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:46,096] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 191.0 (TID 167). 5951 bytes result sent to driver
[2023-03-27T03:15:46,097] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 191.0 (TID 167) in 16 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:46,097] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 191.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:46,098] [INFO ] o.a.s.s.DAGScheduler - ResultStage 191 (foreach at UnboundedDataset.java:79) finished in 0.020 s
[2023-03-27T03:15:46,098] [INFO ] o.a.s.s.DAGScheduler - Job 167 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:46,098] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 191: Stage finished
[2023-03-27T03:15:46,098] [INFO ] o.a.s.s.DAGScheduler - Job 167 finished: foreach at UnboundedDataset.java:79, took 0.022955 s
[2023-03-27T03:15:46,099] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886943735 ms.0 from job set of time 1679886943735 ms
[2023-03-27T03:15:46,099] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.364 s for time 1679886943735 ms (execution: 0.031 s)
[2023-03-27T03:15:46,099] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1888 from persistence list
[2023-03-27T03:15:46,102] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1887 from persistence list
[2023-03-27T03:15:46,102] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1428 from persistence list
[2023-03-27T03:15:46,103] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1427 from persistence list
[2023-03-27T03:15:46,103] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1908 from persistence list
[2023-03-27T03:15:46,104] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1907 from persistence list
[2023-03-27T03:15:46,104] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1906 from persistence list
[2023-03-27T03:15:46,104] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1905 from persistence list
[2023-03-27T03:15:46,104] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1904 from persistence list
[2023-03-27T03:15:46,104] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1903 from persistence list
[2023-03-27T03:15:46,105] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1902 from persistence list
[2023-03-27T03:15:46,105] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1901 from persistence list
[2023-03-27T03:15:46,105] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1900 from persistence list
[2023-03-27T03:15:46,105] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1899 from persistence list
[2023-03-27T03:15:46,106] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1898 from persistence list
[2023-03-27T03:15:46,106] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1897 from persistence list
[2023-03-27T03:15:46,106] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1896 from persistence list
[2023-03-27T03:15:46,106] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1895 from persistence list
[2023-03-27T03:15:46,107] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1894 from persistence list
[2023-03-27T03:15:46,108] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1893 from persistence list
[2023-03-27T03:15:46,108] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1892 from persistence list
[2023-03-27T03:15:46,108] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1891 from persistence list
[2023-03-27T03:15:46,109] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1890 from persistence list
[2023-03-27T03:15:46,109] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1889 from persistence list
[2023-03-27T03:15:46,109] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886943735 ms
[2023-03-27T03:15:46,109] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886943735 ms
[2023-03-27T03:15:46,109] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886943735 ms
[2023-03-27T03:15:46,111] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886943735 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886943735'
[2023-03-27T03:15:46,111] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886943735 ms to writer queue
[2023-03-27T03:15:46,126] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886918710
[2023-03-27T03:15:46,127] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886943735 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886943735', took 22760 bytes and 15 ms
[2023-03-27T03:15:46,127] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886943735 ms
[2023-03-27T03:15:46,127] [INFO ] o.a.s.s.d.DStreamCheckpointData - Deleted checkpoint file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/6a51bdf3-a8f0-4ea3-b694-153664e45b0a/rdd-1428' for time 1679886843635 ms
[2023-03-27T03:15:46,127] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886943735 ms
[2023-03-27T03:15:46,127] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:46,127] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886843635: 
[2023-03-27T03:15:46,128] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886838630 ms
[2023-03-27T03:15:46,133] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:40.779Z, highWatermark=2023-03-27T03:15:45.771Z, synchronizedProcessingTime=2023-03-27T03:15:43.735Z}}
[2023-03-27T03:15:46,133] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886943735 has completed, watermarks have been updated.
[2023-03-27T03:15:48,741] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:48,748] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:48,748] [INFO ] o.a.s.s.DAGScheduler - Got job 168 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:48,748] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 192 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:48,748] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:48,749] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:48,749] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 192 (MapPartitionsRDD[1937] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:48,751] [INFO ] o.a.s.SparkContext - Created broadcast 184 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:48,751] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 192 (MapPartitionsRDD[1937] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:48,751] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 192.0 with 1 tasks resource profile 0
[2023-03-27T03:15:48,752] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 192.0 (TID 168) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 10295 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:48,753] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 192.0 (TID 168)
[2023-03-27T03:15:48,757] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:48,759] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:50,767] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2008 millis on reading.
[2023-03-27T03:15:50,889] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 192.0 (TID 168)
[rdd_1910_0]
[2023-03-27T03:15:50,889] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 192.0 (TID 168). 7847 bytes result sent to driver
[2023-03-27T03:15:50,890] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 192.0 (TID 168) in 2138 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:50,890] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 192.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:50,890] [INFO ] o.a.s.s.DAGScheduler - ResultStage 192 (DStream at SparkUnboundedSource.java:174) finished in 2.140 s
[2023-03-27T03:15:50,891] [INFO ] o.a.s.s.DAGScheduler - Job 168 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:50,891] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 192: Stage finished
[2023-03-27T03:15:50,891] [INFO ] o.a.s.s.DAGScheduler - Job 168 finished: DStream at SparkUnboundedSource.java:174, took 2.143072 s
[2023-03-27T03:15:50,910] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886948740 ms
[2023-03-27T03:15:50,910] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886948740 ms
[2023-03-27T03:15:50,910] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886948740 ms
[2023-03-27T03:15:50,911] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886948740 ms
[2023-03-27T03:15:50,911] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886948740 ms.0 from job set of time 1679886948740 ms
[2023-03-27T03:15:50,912] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886948740 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886948740'
[2023-03-27T03:15:50,913] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886948740 ms to writer queue
[2023-03-27T03:15:50,918] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:50,919] [INFO ] o.a.s.s.DAGScheduler - Got job 169 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:50,919] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 193 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:50,919] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:50,919] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:50,920] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 193 (MapPartitionsRDD[1957] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:50,923] [INFO ] o.a.s.SparkContext - Created broadcast 185 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:50,923] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 193 (MapPartitionsRDD[1957] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:50,923] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 193.0 with 1 tasks resource profile 0
[2023-03-27T03:15:50,924] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 193.0 (TID 169) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 10295 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:50,924] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 193.0 (TID 169)
[2023-03-27T03:15:50,927] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886923715.bk
[2023-03-27T03:15:50,927] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886948740 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886948740', took 22776 bytes and 15 ms
[2023-03-27T03:15:50,932] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:50,938] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 193.0 (TID 169). 5951 bytes result sent to driver
[2023-03-27T03:15:50,939] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 193.0 (TID 169) in 15 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:50,939] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 193.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:50,940] [INFO ] o.a.s.s.DAGScheduler - ResultStage 193 (foreach at UnboundedDataset.java:79) finished in 0.019 s
[2023-03-27T03:15:50,940] [INFO ] o.a.s.s.DAGScheduler - Job 169 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:50,940] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 193: Stage finished
[2023-03-27T03:15:50,940] [INFO ] o.a.s.s.DAGScheduler - Job 169 finished: foreach at UnboundedDataset.java:79, took 0.021568 s
[2023-03-27T03:15:50,941] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886948740 ms.0 from job set of time 1679886948740 ms
[2023-03-27T03:15:50,941] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.201 s for time 1679886948740 ms (execution: 0.030 s)
[2023-03-27T03:15:50,941] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1912 from persistence list
[2023-03-27T03:15:50,942] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1911 from persistence list
[2023-03-27T03:15:50,942] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1453 from persistence list
[2023-03-27T03:15:50,946] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1452 from persistence list
[2023-03-27T03:15:50,947] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1933 from persistence list
[2023-03-27T03:15:50,947] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1932 from persistence list
[2023-03-27T03:15:50,948] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1931 from persistence list
[2023-03-27T03:15:50,948] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1930 from persistence list
[2023-03-27T03:15:50,949] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1929 from persistence list
[2023-03-27T03:15:50,949] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1928 from persistence list
[2023-03-27T03:15:50,950] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1927 from persistence list
[2023-03-27T03:15:50,950] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1926 from persistence list
[2023-03-27T03:15:50,951] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1925 from persistence list
[2023-03-27T03:15:50,951] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1924 from persistence list
[2023-03-27T03:15:50,952] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1923 from persistence list
[2023-03-27T03:15:50,952] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1922 from persistence list
[2023-03-27T03:15:50,952] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1921 from persistence list
[2023-03-27T03:15:50,952] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1920 from persistence list
[2023-03-27T03:15:50,953] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1919 from persistence list
[2023-03-27T03:15:50,953] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1918 from persistence list
[2023-03-27T03:15:50,953] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1917 from persistence list
[2023-03-27T03:15:50,953] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1916 from persistence list
[2023-03-27T03:15:50,954] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1915 from persistence list
[2023-03-27T03:15:50,954] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1914 from persistence list
[2023-03-27T03:15:50,954] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886948740 ms
[2023-03-27T03:15:50,954] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886948740 ms
[2023-03-27T03:15:50,954] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886948740 ms
[2023-03-27T03:15:50,956] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886948740 ms to writer queue
[2023-03-27T03:15:50,956] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886948740 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886948740'
[2023-03-27T03:15:50,969] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886923715
[2023-03-27T03:15:50,969] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886948740 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886948740', took 22772 bytes and 12 ms
[2023-03-27T03:15:50,969] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886948740 ms
[2023-03-27T03:15:50,969] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886948740 ms
[2023-03-27T03:15:50,969] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:50,970] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886848640: 
[2023-03-27T03:15:50,970] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886843635 ms
[2023-03-27T03:15:50,971] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:45.771Z, highWatermark=2023-03-27T03:15:50.767Z, synchronizedProcessingTime=2023-03-27T03:15:48.740Z}}
[2023-03-27T03:15:50,971] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886948740 has completed, watermarks have been updated.
[2023-03-27T03:15:53,745] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:53,757] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:53,757] [INFO ] o.a.s.s.DAGScheduler - Got job 170 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:53,757] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 194 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:53,757] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:53,758] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:53,760] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 194 (MapPartitionsRDD[1961] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:53,762] [INFO ] o.a.s.SparkContext - Created broadcast 186 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:53,763] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 194 (MapPartitionsRDD[1961] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:53,764] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 194.0 with 1 tasks resource profile 0
[2023-03-27T03:15:53,765] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 194.0 (TID 170) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 13896 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:53,765] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 194.0 (TID 170)
[2023-03-27T03:15:53,771] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:53,772] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:55,781] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2008 millis on reading.
[2023-03-27T03:15:55,886] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 194.0 (TID 170)
[rdd_1935_0]
[2023-03-27T03:15:55,886] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 194.0 (TID 170). 7847 bytes result sent to driver
[2023-03-27T03:15:55,887] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 194.0 (TID 170) in 2123 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:55,887] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 194.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:55,887] [INFO ] o.a.s.s.DAGScheduler - ResultStage 194 (DStream at SparkUnboundedSource.java:174) finished in 2.127 s
[2023-03-27T03:15:55,888] [INFO ] o.a.s.s.DAGScheduler - Job 170 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:55,888] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 194: Stage finished
[2023-03-27T03:15:55,888] [INFO ] o.a.s.s.DAGScheduler - Job 170 finished: DStream at SparkUnboundedSource.java:174, took 2.130912 s
[2023-03-27T03:15:55,906] [INFO ] o.a.s.s.s.JobScheduler - Added jobs for time 1679886953745 ms
[2023-03-27T03:15:55,906] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886953745 ms
[2023-03-27T03:15:55,906] [INFO ] o.a.s.s.s.JobScheduler - Starting job streaming job 1679886953745 ms.0 from job set of time 1679886953745 ms
[2023-03-27T03:15:55,906] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886953745 ms
[2023-03-27T03:15:55,906] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886953745 ms
[2023-03-27T03:15:55,908] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886953745 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886953745'
[2023-03-27T03:15:55,908] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886953745 ms to writer queue
[2023-03-27T03:15:55,912] [INFO ] o.a.s.SparkContext - Starting job: foreach at UnboundedDataset.java:79
[2023-03-27T03:15:55,913] [INFO ] o.a.s.s.DAGScheduler - Got job 171 (foreach at UnboundedDataset.java:79) with 1 output partitions
[2023-03-27T03:15:55,913] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 195 (foreach at UnboundedDataset.java:79)
[2023-03-27T03:15:55,913] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:55,913] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:55,914] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 195 (MapPartitionsRDD[1981] at map at TranslationUtils.java:137), which has no missing parents
[2023-03-27T03:15:55,916] [INFO ] o.a.s.SparkContext - Created broadcast 187 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:55,917] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 195 (MapPartitionsRDD[1981] at map at TranslationUtils.java:137) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:55,917] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 195.0 with 1 tasks resource profile 0
[2023-03-27T03:15:55,918] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 195.0 (TID 171) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 13896 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:55,918] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 195.0 (TID 171)
[2023-03-27T03:15:55,920] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886928720.bk
[2023-03-27T03:15:55,920] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886953745 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886953745', took 22766 bytes and 12 ms
[2023-03-27T03:15:55,924] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:55,929] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 195.0 (TID 171). 5951 bytes result sent to driver
[2023-03-27T03:15:55,929] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 195.0 (TID 171) in 12 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:15:55,930] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 195.0, whose tasks have all completed, from pool 
[2023-03-27T03:15:55,930] [INFO ] o.a.s.s.DAGScheduler - ResultStage 195 (foreach at UnboundedDataset.java:79) finished in 0.016 s
[2023-03-27T03:15:55,930] [INFO ] o.a.s.s.DAGScheduler - Job 171 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:15:55,931] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 195: Stage finished
[2023-03-27T03:15:55,931] [INFO ] o.a.s.s.DAGScheduler - Job 171 finished: foreach at UnboundedDataset.java:79, took 0.018300 s
[2023-03-27T03:15:55,931] [INFO ] o.a.s.s.s.JobScheduler - Finished job streaming job 1679886953745 ms.0 from job set of time 1679886953745 ms
[2023-03-27T03:15:55,931] [INFO ] o.a.s.s.s.JobScheduler - Total delay: 2.186 s for time 1679886953745 ms (execution: 0.025 s)
[2023-03-27T03:15:55,932] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1937 from persistence list
[2023-03-27T03:15:55,933] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1936 from persistence list
[2023-03-27T03:15:55,934] [INFO ] o.a.s.s.r.MapWithStateRDD - Removing RDD 1477 from persistence list
[2023-03-27T03:15:55,934] [INFO ] o.a.b.r.s.i.SourceRDD$Unbounded - Removing RDD 1476 from persistence list
[2023-03-27T03:15:55,935] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1957 from persistence list
[2023-03-27T03:15:55,936] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1956 from persistence list
[2023-03-27T03:15:55,937] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1955 from persistence list
[2023-03-27T03:15:55,937] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1954 from persistence list
[2023-03-27T03:15:55,938] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1953 from persistence list
[2023-03-27T03:15:55,938] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1952 from persistence list
[2023-03-27T03:15:55,939] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1951 from persistence list
[2023-03-27T03:15:55,940] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1950 from persistence list
[2023-03-27T03:15:55,940] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1949 from persistence list
[2023-03-27T03:15:55,940] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1948 from persistence list
[2023-03-27T03:15:55,941] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1947 from persistence list
[2023-03-27T03:15:55,941] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1946 from persistence list
[2023-03-27T03:15:55,942] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1945 from persistence list
[2023-03-27T03:15:55,942] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1944 from persistence list
[2023-03-27T03:15:55,942] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1943 from persistence list
[2023-03-27T03:15:55,942] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1942 from persistence list
[2023-03-27T03:15:55,942] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1941 from persistence list
[2023-03-27T03:15:55,943] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1940 from persistence list
[2023-03-27T03:15:55,943] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1939 from persistence list
[2023-03-27T03:15:55,943] [INFO ] o.a.s.r.MapPartitionsRDD - Removing RDD 1938 from persistence list
[2023-03-27T03:15:55,943] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886953745 ms
[2023-03-27T03:15:55,943] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886953745 ms
[2023-03-27T03:15:55,943] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886953745 ms
[2023-03-27T03:15:55,944] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886953745 ms to writer queue
[2023-03-27T03:15:55,945] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886953745 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886953745'
[2023-03-27T03:15:55,959] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886928720
[2023-03-27T03:15:55,959] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder - Put new watermark block: {0=SparkWatermarks{lowWatermark=2023-03-27T03:15:50.767Z, highWatermark=2023-03-27T03:15:55.781Z, synchronizedProcessingTime=2023-03-27T03:15:53.745Z}}
[2023-03-27T03:15:55,959] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886953745 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886953745', took 22762 bytes and 14 ms
[2023-03-27T03:15:55,959] [INFO ] o.a.b.r.s.u.GlobalWatermarkHolder$WatermarkAdvancingStreamingListener - Batch with timestamp: 1679886953745 has completed, watermarks have been updated.
[2023-03-27T03:15:55,959] [INFO ] o.a.s.s.DStreamGraph - Clearing checkpoint data for time 1679886953745 ms
[2023-03-27T03:15:55,959] [INFO ] o.a.s.s.DStreamGraph - Cleared checkpoint data for time 1679886953745 ms
[2023-03-27T03:15:55,959] [INFO ] o.a.s.s.s.ReceivedBlockTracker - Deleting batches: 
[2023-03-27T03:15:55,959] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Attempting to clear 0 old log files in file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/receivedBlockMetadata older than 1679886853645: 
[2023-03-27T03:15:55,960] [INFO ] o.a.s.s.s.InputInfoTracker - remove old batch metadata: 1679886848640 ms
[2023-03-27T03:15:58,750] [INFO ] o.a.b.r.s.i.SourceDStream - Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
[2023-03-27T03:15:58,764] [INFO ] o.a.s.SparkContext - Starting job: DStream at SparkUnboundedSource.java:174
[2023-03-27T03:15:58,765] [INFO ] o.a.s.s.DAGScheduler - Got job 172 (DStream at SparkUnboundedSource.java:174) with 1 output partitions
[2023-03-27T03:15:58,765] [INFO ] o.a.s.s.DAGScheduler - Final stage: ResultStage 196 (DStream at SparkUnboundedSource.java:174)
[2023-03-27T03:15:58,765] [INFO ] o.a.s.s.DAGScheduler - Parents of final stage: List()
[2023-03-27T03:15:58,765] [INFO ] o.a.s.s.DAGScheduler - Missing parents: List()
[2023-03-27T03:15:58,766] [INFO ] o.a.s.s.DAGScheduler - Submitting ResultStage 196 (MapPartitionsRDD[1985] at map at SparkUnboundedSource.java:116), which has no missing parents
[2023-03-27T03:15:58,768] [INFO ] o.a.s.SparkContext - Created broadcast 188 from broadcast at DAGScheduler.scala:1427
[2023-03-27T03:15:58,768] [INFO ] o.a.s.s.DAGScheduler - Submitting 1 missing tasks from ResultStage 196 (MapPartitionsRDD[1985] at map at SparkUnboundedSource.java:116) (first 15 tasks are for partitions Vector(0))
[2023-03-27T03:15:58,768] [INFO ] o.a.s.s.TaskSchedulerImpl - Adding task set 196.0 with 1 tasks resource profile 0
[2023-03-27T03:15:58,770] [INFO ] o.a.s.s.TaskSetManager - Starting task 0.0 in stage 196.0 (TID 172) (8516b817a5d0, executor driver, partition 0, PROCESS_LOCAL, 17497 bytes) taskResourceAssignments Map()
[2023-03-27T03:15:58,771] [INFO ] o.a.s.e.Executor - Running task 0.0 in stage 196.0 (TID 172)
[2023-03-27T03:15:58,782] [INFO ] o.a.b.s.i.a.s.DefaultS3ClientBuilderFactory - The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
[2023-03-27T03:15:58,784] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Continue reading from an existing CheckpointMark.
[2023-03-27T03:15:59,615] [INFO ] o.a.s.s.StreamingContext - Invoking stop(stopGracefully=false) from shutdown hook
[2023-03-27T03:15:59,617] [INFO ] o.a.s.s.u.BatchedWriteAheadLog - BatchedWriteAheadLog shutting down at time: 1679886959617.
[2023-03-27T03:15:59,618] [WARN ] o.a.s.s.u.BatchedWriteAheadLog - BatchedWriteAheadLog Writer queue interrupted.
[2023-03-27T03:15:59,619] [INFO ] o.a.s.s.u.BatchedWriteAheadLog - BatchedWriteAheadLog Writer thread exiting.
[2023-03-27T03:15:59,619] [INFO ] o.t.s.c.c.ContainerManager - Closed container kafka
[2023-03-27T03:15:59,620] [INFO ] o.a.s.s.u.FileBasedWriteAheadLog_ReceivedBlockTracker - Stopped write ahead log manager
[2023-03-27T03:15:59,621] [INFO ] o.a.s.s.s.ReceiverTracker - ReceiverTracker stopped
[2023-03-27T03:15:59,622] [INFO ] o.a.s.s.s.JobGenerator - Stopping JobGenerator immediately
[2023-03-27T03:15:59,623] [INFO ] o.a.s.s.u.RecurringTimer - Stopped timer for JobGenerator after time 1679886958750
[2023-03-27T03:15:59,624] [INFO ] o.t.s.c.c.ContainerManager - Closed container jdbc-ee
[2023-03-27T03:15:59,624] [INFO ] o.t.s.c.r.m.ComponentManager - Released the contextual ComponentManager instance (classloader=jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27, jvm=563784@8516b817a5d0)
[2023-03-27T03:16:00,791] [INFO ] o.a.b.r.s.s.StateSpecFunctions - Source id 0_0 spent 2006 millis on reading.
[2023-03-27T03:16:01,020] [INFO ] o.a.s.e.Executor - 1 block locks were not released by task 0.0 in stage 196.0 (TID 172)
[rdd_1959_0]
[2023-03-27T03:16:01,020] [INFO ] o.a.s.e.Executor - Finished task 0.0 in stage 196.0 (TID 172). 7890 bytes result sent to driver
[2023-03-27T03:16:01,021] [INFO ] o.a.s.s.TaskSetManager - Finished task 0.0 in stage 196.0 (TID 172) in 2252 ms on 8516b817a5d0 (executor driver) (1/1)
[2023-03-27T03:16:01,022] [INFO ] o.a.s.s.TaskSchedulerImpl - Removed TaskSet 196.0, whose tasks have all completed, from pool 
[2023-03-27T03:16:01,022] [INFO ] o.a.s.s.DAGScheduler - ResultStage 196 (DStream at SparkUnboundedSource.java:174) finished in 2.256 s
[2023-03-27T03:16:01,022] [INFO ] o.a.s.s.DAGScheduler - Job 172 is finished. Cancelling potential speculative or zombie tasks for this job
[2023-03-27T03:16:01,022] [INFO ] o.a.s.s.TaskSchedulerImpl - Killing all running tasks in stage 196: Stage finished
[2023-03-27T03:16:01,022] [INFO ] o.a.s.s.DAGScheduler - Job 172 finished: DStream at SparkUnboundedSource.java:174, took 2.257934 s
[2023-03-27T03:16:01,035] [INFO ] o.t.s.c.r.m.ComponentManager - rebuild new component manager
[2023-03-27T03:16:01,041] [INFO ] o.t.s.c.c.ContainerManager - No TALEND-INF/plugins.properties found, will use file resolution
[2023-03-27T03:16:01,043] [INFO ] o.t.s.c.r.m.ComponentManager - ComponentManager version: 1.53.0
[2023-03-27T03:16:01,043] [INFO ] o.t.s.c.r.m.ComponentManager - Creating the contextual ComponentManager instance (classloader=jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27, jvm=563784@8516b817a5d0)
[2023-03-27T03:16:01,043] [INFO ] o.t.s.c.r.m.ComponentManager - Components: []
[2023-03-27T03:16:01,045] [INFO ] o.a.s.s.s.JobGenerator - Checkpointing graph for time 1679886958750 ms
[2023-03-27T03:16:01,045] [INFO ] o.a.s.s.DStreamGraph - Updating checkpoint data for time 1679886958750 ms
[2023-03-27T03:16:01,048] [INFO ] o.a.s.s.DStreamGraph - Updated checkpoint data for time 1679886958750 ms
[2023-03-27T03:16:01,051] [INFO ] o.a.s.s.CheckpointWriter - Submitted checkpoint of time 1679886958750 ms to writer queue
[2023-03-27T03:16:01,051] [INFO ] o.a.s.s.CheckpointWriter - Saving checkpoint for time 1679886958750 ms to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886958750'
[2023-03-27T03:16:01,046] [ERROR] o.a.s.s.s.JobScheduler - Error generating jobs for time 1679886958750 ms
java.lang.IllegalStateException: Shutdown in progress
	at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) ~[?:?]
	at java.lang.Runtime.addShutdownHook(Runtime.java:216) ~[?:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager$SingletonHolder.buildNewComponentManager(ComponentManager.java:243) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager$SingletonHolder.renew(ComponentManager.java:255) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager$SingletonHolder.access$2200(ComponentManager.java:193) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager.lambda$instance$8(ComponentManager.java:557) ~[component-runtime-manager-1.53.0.jar:?]
	at java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:209) ~[?:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager.instance(ComponentManager.java:557) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.datastreams.tck.ConnectorsRepoContainerFinder.find(ConnectorsRepoContainerFinder.java:48) ~[data-processing-runtime-streamsjob-2.23.0.jar:2.23.0]
	at org.talend.sdk.component.runtime.serialization.SerializableService.readResolve(SerializableService.java:37) ~[component-runtime-impl-1.53.0.jar:?]
	at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1136) ~[?:?]
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2220) ~[?:?]
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670) ~[?:?]
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479) ~[?:?]
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373) ~[?:?]
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211) ~[?:?]
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670) ~[?:?]
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:488) ~[?:?]
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:446) ~[?:?]
	at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:75) ~[beam-sdks-java-harness-2.37.0.jar:?]
	at org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:117) ~[beam-sdks-java-harness-2.37.0.jar:?]
	at org.apache.beam.runners.spark.translation.MultiDoFnFunction.<init>(MultiDoFnFunction.java:112) ~[beam-runners-spark-3-2.37.0.jar:?]
	at org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$9.lambda$evaluate$905e84cb$1(StreamingTransformTranslator.java:457) ~[beam-runners-spark-3-2.37.0.jar:?]
	at org.apache.spark.streaming.api.java.JavaDStreamLike.scalaTransform$3(JavaDStreamLike.scala:317) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.api.java.JavaDStreamLike.$anonfun$transformToPair$1(JavaDStreamLike.scala:318) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$transform$2(DStream.scala:668) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$transform$4(DStream.scala:682) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:46) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.FilteredDStream.compute(FilteredDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.$anonfun$compute$1(TransformedDStream.scala:42) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.collection.immutable.List.map(List.scala:293) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:42) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.FilteredDStream.compute(FilteredDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:48) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.DStreamGraph.$anonfun$generateJobs$2(DStreamGraph.scala:123) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:75) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:122) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator.$anonfun$generateJobs$1(JobGenerator.scala:252) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:250) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:186) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:91) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:90) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) [spark-core_2.12-3.2.0.jar:3.2.0]
[2023-03-27T03:16:01,067] [INFO ] o.a.s.s.CheckpointWriter - Deleting file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886933725.bk
[2023-03-27T03:16:01,068] [INFO ] o.a.s.s.CheckpointWriter - Checkpoint for time 1679886958750 ms saved to file 'file:/tmp/pipelineoptionsutil-root-0327030904-b52b9148/spark-checkpoint/checkpoint-1679886958750', took 22762 bytes and 16 ms
[2023-03-27T03:16:01,069] [INFO ] o.a.s.s.CheckpointWriter - CheckpointWriter executor terminated? true, waited for 16 ms.
[2023-03-27T03:16:01,070] [INFO ] o.a.s.s.s.JobGenerator - Stopped JobGenerator
[2023-03-27T03:16:01,072] [INFO ] o.a.s.s.s.JobScheduler - Stopped JobScheduler
[2023-03-27T03:16:01,074] [INFO ] o.a.s.s.StreamingContext - StreamingContext stopped successfully
[2023-03-27T03:16:01,075] [INFO ] o.a.s.SparkContext - Invoking stop() from shutdown hook
[2023-03-27T03:16:01,075] [WARN ] o.a.s.s.StreamingContext - StreamingContext has already been stopped
[2023-03-27T03:16:01,075] [INFO ] o.t.d.b.c.r.ServerSocketAvroSchemaRegistry - Shutdown registry containing {}
[2023-03-27T03:16:01,076] [ERROR] o.t.d.s.FullRunJob$ - DATA_STREAMS_ERROR_MSG:Shutdown in progress
[2023-03-27T03:16:01,076] [ERROR] o.t.d.s.FullRunJob$ - Original Throwable:
java.lang.IllegalStateException: Shutdown in progress
	at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) ~[?:?]
	at java.lang.Runtime.addShutdownHook(Runtime.java:216) ~[?:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager$SingletonHolder.buildNewComponentManager(ComponentManager.java:243) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager$SingletonHolder.renew(ComponentManager.java:255) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager$SingletonHolder.access$2200(ComponentManager.java:193) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager.lambda$instance$8(ComponentManager.java:557) ~[component-runtime-manager-1.53.0.jar:?]
	at java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:209) ~[?:?]
	at org.talend.sdk.component.runtime.manager.ComponentManager.instance(ComponentManager.java:557) ~[component-runtime-manager-1.53.0.jar:?]
	at org.talend.datastreams.tck.ConnectorsRepoContainerFinder.find(ConnectorsRepoContainerFinder.java:48) ~[data-processing-runtime-streamsjob-2.23.0.jar:2.23.0]
	at org.talend.sdk.component.runtime.serialization.SerializableService.readResolve(SerializableService.java:37) ~[component-runtime-impl-1.53.0.jar:?]
	at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1136) ~[?:?]
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2220) ~[?:?]
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670) ~[?:?]
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479) ~[?:?]
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373) ~[?:?]
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211) ~[?:?]
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670) ~[?:?]
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:488) ~[?:?]
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:446) ~[?:?]
	at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:75) ~[beam-sdks-java-harness-2.37.0.jar:?]
	at org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:117) ~[beam-sdks-java-harness-2.37.0.jar:?]
	at org.apache.beam.runners.spark.translation.MultiDoFnFunction.<init>(MultiDoFnFunction.java:112) ~[beam-runners-spark-3-2.37.0.jar:?]
	at org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$9.lambda$evaluate$905e84cb$1(StreamingTransformTranslator.java:457) ~[beam-runners-spark-3-2.37.0.jar:?]
	at org.apache.spark.streaming.api.java.JavaDStreamLike.scalaTransform$3(JavaDStreamLike.scala:317) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.api.java.JavaDStreamLike.$anonfun$transformToPair$1(JavaDStreamLike.scala:318) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$transform$2(DStream.scala:668) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$transform$4(DStream.scala:682) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:46) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.FilteredDStream.compute(FilteredDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.$anonfun$compute$1(TransformedDStream.scala:42) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.collection.immutable.List.map(List.scala:293) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:42) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.FilteredDStream.compute(FilteredDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:36) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$3(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$2(DStream.scala:343) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:417) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.DStream.$anonfun$getOrCompute$1(DStream.scala:342) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:335) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:48) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.DStreamGraph.$anonfun$generateJobs$2(DStreamGraph.scala:123) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:75) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:122) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator.$anonfun$generateJobs$1(JobGenerator.scala:252) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:250) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:186) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:91) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:90) ~[spark-streaming_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) ~[spark-core_2.12-3.2.0.jar:3.2.0]
[2023-03-27T03:16:01,089] [INFO ] o.a.s.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
[2023-03-27T03:16:01,116] [INFO ] o.a.s.s.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
[2023-03-27T03:16:01,127] [INFO ] o.a.s.SparkContext - Successfully stopped SparkContext
[2023-03-27T03:16:01,128] [INFO ] o.a.s.u.ShutdownHookManager - Shutdown hook called
[2023-03-27T03:16:01,128] [INFO ] o.a.s.u.ShutdownHookManager - Deleting directory /tmp/spark-9db8fc52-5bdf-402b-a27b-c4830a5cbdfc