8000 HDFS-16521. DFS API to retrieve slow datanodes (#4107) by virajjasani · Pull Request #4259 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HDFS-16521. DFS API to retrieve slow datanodes (#4107) #4259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 5, 2022
Merged

HDFS-16521. DFS API to retrieve slow datanodes (#4107) #4259

merged 2 commits into from
May 5, 2022

Conversation

virajjasani
Copy link
Contributor

branch-3.3 backport PR of #4107

@virajjasani virajjasani mentioned this pull request May 3, 2022
Merged
1 task
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 10m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 46s Maven dependency ordering for branch
+1 💚 mvninstall 27m 8s branch-3.3 passed
+1 💚 compile 4m 2s branch-3.3 passed
+1 💚 checkstyle 1m 18s branch-3.3 passed
+1 💚 mvnsite 3m 28s branch-3.3 passed
+1 💚 javadoc 3m 34s branch-3.3 passed
+1 💚 spotbugs 7m 46s branch-3.3 passed
+1 💚 shadedclient 26m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for patch
+1 💚 mvninstall 2m 51s the patch passed
+1 💚 compile 3m 46s the patch passed
+1 💚 cc 3m 46s the patch passed
-1 ❌ javac 3m 46s /results-compile-javac-hadoop-hdfs-project.txt hadoop-hdfs-project generated 1 new + 746 unchanged - 0 fixed = 747 total (was 746)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 2s hadoop-hdfs-project: The patch generated 0 new + 470 unchanged - 1 fixed = 470 total (was 471)
+1 💚 mvnsite 2m 54s the patch passed
+1 💚 javadoc 2m 54s the patch passed
+1 💚 spotbugs 7m 37s the patch passed
+1 💚 shadedclient 26m 2s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 16s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 207m 47s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 15m 57s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 58s The patch does not generate ASF License warnings.
376m 41s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.cli.TestHDFSCLI
hadoop.hdfs.TestRollingUpgrade
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/1/artifact/out/Dockerfile
GITHUB PR #4259
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat markdownlint
uname Linux 1a4d0d83a476 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / ae723bc
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/1/testReport/
Max. process+thread count 2050 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 buf 0m 0s buf was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 23s Maven dependency ordering for branch
+1 💚 mvninstall 23m 41s branch-3.3 passed
+1 💚 compile 3m 57s branch-3.3 passed
+1 💚 checkstyle 1m 26s branch-3.3 passed
+1 💚 mvnsite 3m 54s branch-3.3 passed
+1 💚 javadoc 4m 3s branch-3.3 passed
+1 💚 spotbugs 7m 48s branch-3.3 passed
+1 💚 shadedclient 25m 2s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for patch
+1 💚 mvninstall 3m 1s the patch passed
+1 💚 compile 3m 43s the patch passed
+1 💚 cc 3m 43s the patch passed
-1 ❌ javac 3m 43s /results-compile-javac-hadoop-hdfs-project.txt hadoop-hdfs-project generated 1 new + 746 unchanged - 0 fixed = 747 total (was 746)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 3s hadoop-hdfs-project: The patch generated 0 new + 470 unchanged - 1 fixed = 470 total (was 471)
+1 💚 mvnsite 3m 5s the patch passed
+1 💚 javadoc 3m 10s the patch passed
+1 💚 spotbugs 7m 24s the patch passed
+1 💚 shadedclient 24m 39s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 27s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 188m 45s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 15m 38s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 14s The patch does not generate ASF License warnings.
348m 56s
Reason Tests
Failed junit tests hadoop.hdfs.TestFileCreation
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.cli.TestHDFSCLI
hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.server.blockmanagement.TestAvailableSpaceRackFaultTolerantBPP
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpc
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/3/artifact/out/Dockerfile
GITHUB PR #4259
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat markdownlint
uname Linux c588499bee11 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / ae723bc
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/3/testReport/
Max. process+thread count 3003 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/3/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 14m 56s Maven dependency ordering for branch
+1 💚 mvninstall 26m 27s branch-3.3 passed
+1 💚 compile 3m 57s branch-3.3 passed
+1 💚 checkstyle 1m 20s branch-3.3 passed
+1 💚 mvnsite 3m 28s branch-3.3 passed
+1 💚 javadoc 3m 32s branch-3.3 passed
+1 💚 spotbugs 7m 49s branch-3.3 passed
+1 💚 shadedclient 26m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for patch
+1 💚 mvninstall 2m 52s the patch passed
+1 💚 compile 3m 47s the patch passed
+1 💚 cc 3m 47s the patch passed
-1 ❌ javac 3m 47s /results-compile-javac-hadoop-hdfs-project.txt hadoop-hdfs-project generated 1 new + 746 unchanged - 0 fixed = 747 total (was 746)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 1s hadoop-hdfs-project: The patch generated 0 new + 470 unchanged - 1 fixed = 470 total (was 471)
+1 💚 mvnsite 2m 51s the patch passed
+1 💚 javadoc 2m 52s the patch passed
+1 💚 spotbugs 7m 35s the patch passed
+1 💚 shadedclient 26m 5s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 18s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 208m 47s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 15m 45s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
365m 39s
Reason Tests
Failed junit tests hadoop.cli.TestHDFSCLI
hadoop.hdfs.TestDecommissionWithBackoffMonitor
hadoop.hdfs.server.blockmanagement.TestBlockManager
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/2/artifact/out/Dockerfile
GITHUB PR #4259
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat markdownlint
uname Linux f5439c5e6ce8 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / ae723bc
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/2/testReport/
Max. process+thread count 2115 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 buf 0m 0s buf was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 44s Maven dependency ordering for branch
+1 💚 mvninstall 23m 43s branch-3.3 passed
+1 💚 compile 3m 58s branch-3.3 passed
+1 💚 checkstyle 1m 26s branch-3.3 passed
+1 💚 mvnsite 3m 55s branch-3.3 passed
+1 💚 javadoc 4m 1s branch-3.3 passed
+1 💚 spotbugs 7m 44s branch-3.3 passed
+1 💚 shadedclient 24m 44s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 3m 0s the patch passed
+1 💚 compile 3m 41s the patch passed
+1 💚 cc 3m 41s the patch passed
-1 ❌ javac 3m 41s /results-compile-javac-hadoop-hdfs-project.txt hadoop-hdfs-project generated 1 new + 746 unchanged - 0 fixed = 747 total (was 746)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 2s hadoop-hdfs-project: The patch generated 0 new + 470 unchanged - 1 fixed = 470 total (was 471)
+1 💚 mvnsite 3m 3s the patch passed
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 3m 10s the patch passed
+1 💚 spotbugs 7m 22s the patch passed
+1 💚 shadedclient 24m 21s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 30s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 196m 54s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 15m 38s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 17s The patch does not generate ASF License warnings.
350m 22s
Reason Tests
Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpc
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/5/artifact/out/Dockerfile
GITHUB PR #4259
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat markdownlint xml
uname Linux cc4e016a4963 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f289ae6
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/5/testReport/
Max. process+thread count 2943 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/5/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 2s Maven dependency ordering for branch
+1 💚 mvninstall 26m 30s branch-3.3 passed
+1 💚 compile 3m 58s branch-3.3 passed
+1 💚 checkstyle 1m 19s branch-3.3 passed
+1 💚 mvnsite 3m 29s branch-3.3 passed
+1 💚 javadoc 3m 33s branch-3.3 passed
+1 💚 spotbugs 7m 42s branch-3.3 passed
+1 💚 shadedclient 26m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for patch
+1 💚 mvninstall 2m 50s the patch passed
+1 💚 compile 3m 52s the patch passed
+1 💚 cc 3m 52s the patch passed
-1 ❌ javac 3m 52s /results-compile-javac-hadoop-hdfs-project.txt hadoop-hdfs-project generated 1 new + 746 unchanged - 0 fixed = 747 total (was 746)
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 1m 2s hadoop-hdfs-project: The patch generated 0 new + 470 unchanged - 1 fixed = 470 total (was 471)
+1 💚 mvnsite 2m 53s the patch passed
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 javadoc 2m 53s the patch passed
+1 💚 spotbugs 7m 35s the patch passed
+1 💚 shadedclient 26m 32s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 18s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 209m 42s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 15m 52s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
367m 35s
Reason Tests
Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestRollingUpgrade
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/6/artifact/out/Dockerfile
GITHUB PR #4259
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat markdownlint xml
uname Linux 55f866b639bd 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f289ae6
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/6/testReport/
Max. process+thread count 2226 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4259/6/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@jojochuang
Copy link
Contributor

The failed tests do not look relevant. Some of them seem to be failing since before.

@jojochuang jojochuang merged commit 4d935ea into apache:branch-3.3 May 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@virajjasani @hadoop-yetus @jojochuang
0