Description
The spark implementation of the HDFS Beam components are hopelessly broken in 0.3.0-incubating. They will not work in a Spark runner.
The next release 0.4.0-incubating is supposed to fix this problem by hot-swapping the broken HDFSFileSink with a Spark appropriate implementation.
We need to upgrade our version of Beam API.