Loading...

Apply template

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 0.20.0
Affects Version/s: None
Component/s: None
Labels:
None

Steps to reproduce:
Hide

Use OS Windows

Check that HADOOP_HOME env var is not set in your OS

cd simplefileio-runtime maven module

remove Ignore annotation from S3DatasetRuntimeTestIT class

run following command

mvn clean test-compile failsafe:integration-test -Dit.test=S3DatasetRuntimeTestIT

Test will fail with described exception
Show
Use OS Windows Check that HADOOP_HOME env var is not set in your OS cd simplefileio-runtime maven module remove Ignore annotation from S3DatasetRuntimeTestIT class run following command mvn clean test-compile failsafe:integration-test -Dit.test=S3DatasetRuntimeTestIT Test will fail with described exception
Platform:
All
Project Size:
Small
Found in Release:

0.19.0

Description

S3 component from component/components-fileio module requires following things:

winutils.exe
hadoop.dll
HADOOP_HOME env var to be set
%HADOOP_HOME%\bin should be appended to PATH env variable

If above is missed following exception occurs on Windows OS:

ERROR [main] (Shell.java:374) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
        at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
        at org.talend.components.simplefileio.runtime.s3.S3Connection.createFileSystem(S3Connection.java:59)
        at org.talend.components.simplefileio.runtime.s3.S3TestResource.after(S3TestResource.java:179)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:351)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:245)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:184)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:312)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:266)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:147)
 WARN [main] (NativeCodeLoader.java:62) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Solution for TCOMP service is to pack service with above binaries and to set env var in start.bat file

Attachments

Activity

People

Assignee:: Ivan Gonchar

Reporter:: Ivan Gonchar

Fixed by:: Ivan Gonchar

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 03/Jul/17 11:00 AM

Updated:: 04/Jun/20 12:20 PM

Resolved:: 11/Jul/17 2:09 PM

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

1d 7h