Details
-
Backlog Task
-
Resolution: Fixed
-
Minor
-
None
-
None
-
None
-
Small
Description
A component should be able to provide runtime metrics to the job (counters like "how many records did I process and/or reject?".
Beam has a limited facility to do counters called {{Aggregator}}s, which are only available in {{PTransform}}s based on {{DoFn}}s.
More advanced counters can always be created by adding more {{PTransform}}s to a job (counter and aggregation steps).
We should investigate the most efficient way to add metrics and monitoring to running and finished TCOMP jobs.