We've hit the problem at this line again:
In brief, to determine whether a class is one of the Beam core classes, it tests against the jar name. Unfortunately, Databricks renames jar files, and beam-sdks-java-core- becomes:
/tmp/addedFile3061440985423865599dbfs__tpd_staging_datastreams_deps_org_apache_beam_beam_sdks_java_core_2_8_0_beam_sdks_java_core_2_8-d79c3.jar
Both the prefix (addedFile3016....) and suffix (-d79c3.jar) can change between runs.