HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters #2605

fengnanli · 2021-01-07T22:38:43Z

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

.../src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterWebHdfsMethods.java

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

hadoop-yetus · 2021-01-08T00:53:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	40m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 26s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 31s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 38s		trunk passed
+1 💚	shadedclient	18m 6s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 13s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 26s		the patch passed
-0 ⚠️	checkstyle	0m 16s	/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 48s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 47s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
-1 ❌	findbugs	1m 16s	/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
			_ Other Tests _
-1 ❌	unit	9m 19s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		133m 31s

Reason	Tests
FindBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	Possible null pointer dereference of dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:[line 472]
Failed junit tests	hadoop.fs.contract.router.web.TestRouterWebHDFSContractMkdir
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractDelete
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractRename
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractOpen

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/1/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c5bd54950877 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b612c31`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/1/testReport/
Max. process+thread count	2905 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

fengnanli · 2021-01-10T07:52:48Z

Thanks for the review @goiri. I may need to fix some other unit tests.

hadoop-yetus · 2021-01-10T09:19:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 48s		trunk passed
+1 💚	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 39s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 28s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	shadedclient	16m 37s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 55s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 15s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 14s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 28s		the patch passed
-0 ⚠️	checkstyle	0m 18s	/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 15 unchanged - 0 fixed = 18 total (was 15)
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 26s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 36s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 53s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
-1 ❌	findbugs	1m 18s	/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
			_ Other Tests _
-1 ❌	unit	9m 35s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		89m 2s

Reason	Tests
FindBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	Possible null pointer dereference of dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:[line 473]
Failed junit tests	hadoop.fs.contract.router.web.TestRouterWebHDFSContractSeek
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractRename
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractOpen
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractConcat
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/2/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7c0dafe769c6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ec22850`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/2/testReport/
Max. process+thread count	3295 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

.../src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterWebHdfsMethods.java

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

.../src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterWebHdfsMethods.java

fengnanli · 2021-01-11T19:56:09Z

@goiri Thanks for the further review.
I found these TestRouterWebHdfsContractXX are flaky with my change and I think they are related with the fact that for the MiniRouterDFSCluster these tests created (in RouterWebHDFSContract), datanodes are shared among nameservices. The datanode report would be sometimes overwritten as : ns0->/ns0/rack would be overwritten by ns1->/ns1/rack for the same dn. Therefore for some tests we saw that no available datanode was found.
Do you think we should fix it first? If so should it be in a separate patch? Thanks!

goiri · 2021-01-11T19:58:10Z

@goiri Thanks for the further review.
I found these TestRouterWebHdfsContractXX are flaky with my change and I think they are related with the fact that for the MiniRouterDFSCluster these tests created (in RouterWebHDFSContract), datanodes are shared among nameservices. The datanode report would be sometimes overwritten as : ns0->/ns0/rack would be overwritten by ns1->/ns1/rack for the same dn. Therefore for some tests we saw that no available datanode was found.
Do you think we should fix it first? If so should it be in a separate patch? Thanks!

Adding flaky tests is not a good idea.
Let's try to fix it in this PR.

hadoop-yetus · 2021-01-11T21:46:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 1s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 38s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	18m 27s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 20s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 18s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 29s		the patch passed
-0 ⚠️	checkstyle	0m 18s	/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 15 unchanged - 0 fixed = 18 total (was 15)
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	17m 51s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 49s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
-1 ❌	findbugs	1m 20s	/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
			_ Other Tests _
-1 ❌	unit	9m 20s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		95m 44s

Reason	Tests
FindBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	Possible null pointer dereference of dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:[line 474]
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterWebHdfsMethods
	hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/3/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 73a0537c09a6 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `32abc0a`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/3/testReport/
Max. process+thread count	2415 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-01-12T08:48:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 9s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 27s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	shadedclient	16m 28s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 55s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 14s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 33s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 28s		the patch passed
+1 💚	checkstyle	0m 17s		the patch passed
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	14m 58s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 35s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
-1 ❌	findbugs	1m 17s	/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
			_ Other Tests _
+1 💚	unit	9m 26s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		89m 34s

Reason	Tests
FindBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	Possible null pointer dereference of dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:dns in org.apache.hadoop.hdfs.server.federation.router.RouterWebHdfsMethods.chooseDatanode(Router, String, HttpOpParam$Op, long, String) on exception path Dereferenced at RouterWebHdfsMethods.java:[line 473]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/4/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2d0782c31714 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ca7dd5f`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/4/testReport/
Max. process+thread count	3132 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-01-12T18:41:37Z

Any chance we can fix the findbugs?

fengnanli · 2021-01-12T19:36:06Z

Any chance we can fix the findbugs?

Yeah, uploaded another small change for it. Hopefully it should cover everything.

hadoop-yetus · 2021-01-12T21:05:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 0s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 28s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	shadedclient	16m 41s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 56s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 13s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 33s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 29s		the patch passed
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 3s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 35s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 54s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	1m 15s		the patch passed
			_ Other Tests _
-1 ❌	unit	9m 21s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		88m 39s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterWebHdfsMethods

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/5/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f1c7082c2aa7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `724edb0`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/5/testReport/
Max. process+thread count	3307 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/5/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-01-12T21:55:17Z

Thanks @fengnanli for the fix.
This looks good to me.
@ayushtkn, do you mind double checking?

hadoop-yetus · 2021-01-13T00:40:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 48s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 36s		trunk passed
+1 💚	shadedclient	17m 56s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 43s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 14s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 13s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 27s		the patch passed
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 23s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 48s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	1m 16s		the patch passed
			_ Other Tests _
+1 💚	unit	9m 26s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		93m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/6/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3830f02f4d18 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `724edb0`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/6/testReport/
Max. process+thread count	3073 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/6/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2021-01-14T13:09:47Z

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

+      resolvedNs = rpcServer.getLocationsForPath(path, true)
+          .get(0).getNameserviceId();


Thanx @fengnanli, Seems I have got little confused here -
This method will get call for (OPEN/APPEND/GETFILECHECKSUM) also, There is a check below as well for these operations.
So, Here we are taking the first nameservice as the resolvedNs, and datanodes from other namespace we add into excluded nodes. What I understood by this check below -
if (collection.contains(dn.getName()) || !ns.equals(resolvedNs)) { excludes.add(dn);
So, In case the file was in the other namespace, rather than being in the first one in case of say RANDOM or SPACE order, so will we not put those datanodes into the excluded ones? and apparently the call may fail for say APPEND or OPEN.
Am I missing some check here? if not, the resolvedNs for these cases we should get from the getFileInfo?

Secondly, In case of create call, we are taking the datanodes from the first namespace, are we sure the create call was to the first Namespace only?

Thanks for the review @ayushtkn
For 1, I agree that use getFileInfo is more accurate so I add another filter to limit only CREATE will apply the resolvedNS.
For 2, the first one should be enough since it is the destination with the highest priority based on the destination order.

Actually For 2, I found there is a getCreateLocation inside RouterRpcServer to get a unique create ns. I will use that to make create in parity between rpc and http.

hadoop-yetus · 2021-01-14T22:08:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	59m 1s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	shadedclient	18m 22s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 36s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 16s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 14s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 27s		the patch passed
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 46s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 48s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	1m 15s		the patch passed
			_ Other Tests _
+1 💚	unit	9m 25s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		117m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/7/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux dfcc2bdee034 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `630f8dd`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/7/testReport/
Max. process+thread count	3060 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/7/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

goiri · 2021-01-14T22:33:25Z

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

+        getTrimmedStringCollection(excludeDatanodes);
+    for (DatanodeInfo dn : dns) {
+      String ns = getNsFromDataNodeNetworkLocation(dn.getNetworkLocation());
+      if (collection.contains(dn.getName()) ||


This if starts getting complex, maybe we want to extract it and do multiple levels of ifs there.

I made it if () else if () so the logic is clearer. How do you think?

.../src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterWebHdfsMethods.java

hadoop-yetus · 2021-01-18T00:31:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 31s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	17m 54s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 14s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 27s		the patch passed
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 58s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 39s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 48s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	1m 15s		the patch passed
			_ Other Tests _
+1 💚	unit	9m 42s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		94m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/8/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux aa0dce49a46c 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `97f843d`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/8/testReport/
Max. process+thread count	3064 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/8/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

fengnanli · 2021-02-14T07:13:03Z

@goiri Addressed the comments as suggested. Can you give it another look? Thanks very much!

hadoop-yetus · 2021-02-14T09:16:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	25m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 5s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 31s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	17m 4s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 35s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 52s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	1m 13s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 11s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 27s		the patch passed
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 9s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 49s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	1m 15s		the patch passed
			_ Other Tests _
-1 ❌	unit	16m 59s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		122m 36s

Reason	Tests
Failed junit tests	hadoop.fs.contract.router.TestRouterHDFSContractCreate
	hadoop.fs.contract.router.TestRouterHDFSContractCreateSecure
	hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/9/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux abbb707f18e4 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c3134ab`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/9/testReport/
Max. process+thread count	2232 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/9/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

fengnanli · 2021-02-15T19:11:58Z

Rebase on latest trunk and force push.

hadoop-yetus · 2021-02-15T20:50:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 25s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 31s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	16m 31s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 36s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 53s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	1m 14s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 26s		the patch passed
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 33s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 49s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	1m 18s		the patch passed
			_ Other Tests _
-1 ❌	unit	17m 13s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		97m 49s

Reason	Tests
Failed junit tests	hadoop.fs.contract.router.TestRouterHDFSContractCreate
	hadoop.fs.contract.router.TestRouterHDFSContractCreateSecure

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/10/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c72969321a27 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `07a4220`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/10/testReport/
Max. process+thread count	2225 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/10/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

fengnanli · 2021-02-16T01:12:06Z

The test failures are related with the change in HADOOP-13327 and is under fix in HDFS-15836

.../src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterWebHdfsMethods.java

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java

hadoop-yetus · 2021-02-17T02:34:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 0s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	16m 28s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 36s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 52s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	1m 15s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 27s		the patch passed
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 6s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 49s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	1m 13s		the patch passed
			_ Other Tests _
+1 💚	unit	17m 4s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		97m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/11/artifact/out/Dockerfile
GITHUB PR	#2605
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2622a83dfef0 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9882a22`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/11/testReport/
Max. process+thread count	2226 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2605/11/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

fengnanli · 2021-04-09T05:38:42Z

@goiri Can we land this one? Thanks.

goiri · 2021-04-09T21:48:07Z

If @ayushtkn doesn't have further comments, I'll go ahead and merge this.

ayushtkn

hey @goiri,
please proceed, changes looks good

…ub-clusters (#2605)" This reverts commit cb3ed32.

…ub-clusters (#2605)" (#2900) This reverts commit cb3ed32.

…ers (apache#2605)

…ub-clusters (apache#2605)" (apache#2900) This reverts commit cb3ed32.

goiri changed the title ~~HDFS-15423 [RBF] WebHDFS create file in the resolved nameservice~~ HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters Jan 8, 2021

goiri reviewed Jan 8, 2021

View reviewed changes

goiri approved these changes Jan 11, 2021

View reviewed changes

ayushtkn reviewed Jan 14, 2021

View reviewed changes

goiri approved these changes Jan 14, 2021

View reviewed changes

fengnanli added 9 commits February 15, 2021 11:09

HDFS-15423 [RBF] WebHDFS create file in the resolved nameservice

297652d

Address comments

167d2fc

More fixes

bd3a7bb

Fix all flaky tests

98c69ac

Fix initialization for variables

862d0e6

Fix log

d6e854c

More comments

636c5f7

Address comments

0dcd395

Address comments

e413f34

fengnanli force-pushed the HDFS-15423 branch from 27f21f9 to e413f34 Compare February 15, 2021 19:11

goiri approved these changes Feb 16, 2021

View reviewed changes

.../src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterWebHdfsMethods.java Show resolved Hide resolved

...-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java Show resolved Hide resolved

Address comments

91053da

ayushtkn approved these changes Apr 10, 2021

View reviewed changes

goiri merged commit cb3ed32 into apache:trunk Apr 12, 2021

goiri added a commit that referenced this pull request Apr 13, 2021

Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all s…

be6a9b0

…ub-clusters (#2605)" This reverts commit cb3ed32.

goiri mentioned this pull request Apr 13, 2021

Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters" #2900

Merged

goiri added a commit that referenced this pull request Apr 13, 2021

Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all s…

8246273

…ub-clusters (#2605)" (#2900) This reverts commit cb3ed32.

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clust…

d6b5547

…ers (apache#2605)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all s…

4e44bfc

…ub-clusters (apache#2605)" (apache#2900) This reverts commit cb3ed32.

		resolvedNs = rpcServer.getLocationsForPath(path, true)
		.get(0).getNameserviceId();

HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters #2605

HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters #2605

Uh oh!

Conversation

NOTICE

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!