Uploaded image for project: 'Talend Component Kit'
  1. Talend Component Kit
  2. TCOMP-1407

Databricks: interface javax.json.stream.JsonGeneratorFactory is not visible from class loader

Apply templateInsert Lucidchart DiagramXMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • 1.1.9
    • None
    • None
    • All
    • Small

      When running a Spark job on a Databricks spark-managed cluster, the component manager can't be instantiated due to the above error.

      At the point of creating the enriched java proxy for the JsonGeneratorFactory, the thread context classloader is an instance of org.apache.spark.repl.ExecutorClassLoader

      The JsonGeneratorFactory is loaded by a org.apache.spark.util.MutableURLClassLoader with the container manager classpath.

      Since they don't correspond, the logic here ends up using the system classloader:

      https://github.com/Talend/component-runtime/blob/component-runtime-1.1.6/component-runtime-manager/src/main/java/org/talend/sdk/component/runtime/manager/proxy/JavaProxyEnricherFactory.java#L51

      (which does not have the required JsonGeneratorFactory class at its disposition, causing the above error).

      I'm not sure whether this is a consequence of running in Spark standalone, or because Databricks jobs are launched through a notebook-like API (noting the `repl` in the executor thread context classloader), but it breaks the tacokit.

      The following snippet is a bad workaround, but works on Databricks:

          private ClassLoader selectLoader(final Class[] api, final ClassLoader loader) {
      
              if ("org.apache.spark.repl.ExecutorClassLoader".equals(loader.getClass().getCanonicalName())) {
                  return JavaProxyEnricherFactory.class.getClassLoader();
              }
      
              return Stream.of(api).anyMatch(t -> t.getClassLoader() == loader) || loader.getParent() == null
                      || loader == getSystemClassLoader() ? loader : loader.getParent();
          }
      

            rmannibucau Romain Manni-Bucau
            rskraba Ryan Skraba
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: