8000 HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful f… by fateh288 · Pull Request #6677 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful f… #6677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 18, 2024
Merged

HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful f… #6677

merged 3 commits into from
Jun 18, 2024

Conversation

fateh288
Copy link
Contributor
@fateh288 fateh288 commented Mar 26, 2024 •
edited
Loading

…or testing auth frameworks such as Ranger

Description of PR

The NNThroughputBenchmark can only be used with hdfs user or any user with super user privileges since entering/exiting safemode is a privileged operation. However, when using super user, ACL checks are skipped. Hence it renders the tool to be useless when testing namenode performance along with authorization frameworks such as Apache Ranger / any other authorization frameworks.

An optional argument such as -nonSuperUser can be used to skip the statements such as entering / exiting safemode. This optional argument makes the tool useful for incorporating authorization frameworks into the performance estimation flows.

How was this patch tested?

  1. Unit tests defined in TestNNThroughputBenchmark worked after the patch was applied
  2. Added new unit test case
  3. Tried some commands such to open files which worked both with and without -nonSuperUser option (in a cluster)
    e.g. hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op open -threads 100 -files 100000 -filesPerDir 10000 -keepResults [-nonSuperUser]

For code changes:

  • [Y ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • [ N/A] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [ N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [ N/A] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

dineshchitlangia
dineshchitlangia approved these changes Mar 26, 2024
View reviewed changes
Copy link
Contributor
@dineshchitlangia dineshchitlangia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, pending CI

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 59s Maven dependency ordering for branch
+1 💚 mvninstall 19m 54s trunk passed
+1 💚 compile 8m 55s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 8m 4s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 checkstyle 2m 5s trunk passed
+1 💚 mvnsite 1m 52s trunk passed
+1 💚 javadoc 1m 30s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 53s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 3m 18s trunk passed
+1 💚 shadedclient 21m 24s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for patch
+1 💚 mvninstall 1m 12s the patch passed
+1 💚 compile 8m 34s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 8m 34s the patch passed
+1 💚 compile 8m 14s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 javac 8m 14s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 2m 6s /results-checkstyle-root.txt root: The patch generated 20 new + 113 unchanged - 4 fixed = 133 total (was 117)
+1 💚 mvnsite 1m 51s the patch passed
+1 💚 javadoc 1m 28s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 55s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 3m 36s the patch passed
+1 💚 shadedclient 21m 50s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 38s hadoop-common in the patch passed.
+1 💚 unit 197m 42s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 46s The patch does not generate ASF License warnings.
351m 43s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/1/artifact/out/Dockerfile
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname Linux ed4364f301bc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / d7944e9
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/1/testReport/
Max. process+thread count 4976 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@fateh288 fateh288 force-pushed the feature/HDFS-17439 branch from d7944e9 to e6dd85d Compare March 26, 2024 23:57
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 44s Maven dependency ordering for branch
+1 💚 mvninstall 19m 50s trunk passed
+1 💚 compile 8m 51s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 8m 6s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 checkstyle 2m 7s trunk passed
+1 💚 mvnsite 1m 53s trunk passed
+1 💚 javadoc 1m 31s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 54s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 3m 14s trunk passed
+1 💚 shadedclient 21m 45s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 1m 9s the patch passed
+1 💚 compile 8m 31s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 8m 31s the patch passed
+1 💚 compile 8m 7s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 javac 8m 7s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 4s /results-checkstyle-root.txt root: The patch generated 20 new + 113 unchanged - 4 fixed = 133 total (was 117)
+1 💚 mvnsite 1m 45s the patch passed
+1 💚 javadoc 1m 25s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 54s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 3m 32s the patch passed
+1 💚 shadedclient 21m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 17s hadoop-common in the patch passed.
-1 ❌ unit 200m 31s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 49s The patch does not generate ASF License warnings.
353m 53s
Reason Tests
Failed junit tests hadoop.hdfs.tools.TestDFSAdmin
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/2/artifact/out/Dockerfile
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname Linux 30c762afd2ce 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e6dd85d
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/2/testReport/
Max. process+thread count 4197 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@fateh288
Copy link
Contributor Author

The unit test failures are unrelated to the patch proposed.
@dineshchitlangia can you please propose next steps.

dineshchitlangia
dineshchitlangia requested changes Mar 27, 2024
View reviewed changes
Copy link
Contributor
@dineshchitlangia dineshchitlangia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fateh288 Pls address checkstyle violations generated by your PR.

@fateh288 fateh288 force-pushed the feature/HDFS-17439 branch from 9ee803d to ba26e23 Compare March 27, 2024 21:23
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 9s Maven dependency ordering for branch
+1 💚 mvninstall 19m 53s trunk passed
+1 💚 compile 9m 0s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 8m 15s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 checkstyle 2m 9s trunk passed
+1 💚 mvnsite 1m 52s trunk passed
+1 💚 javadoc 1m 30s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 52s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 3m 22s trunk passed
+1 💚 shadedclient 21m 12s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 1m 11s the patch passed
+1 💚 compile 8m 36s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 8m 36s the patch passed
+1 💚 compile 8m 6s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 javac 8m 6s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 4s /results-checkstyle-root.txt root: The patch generated 6 new + 113 unchanged - 4 fixed = 119 total (was 117)
+1 💚 mvnsite 1m 49s the patch passed
+1 💚 javadoc 1m 30s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 49s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 3m 37s the patch passed
+1 💚 shadedclient 21m 49s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 1s hadoop-common in the patch passed.
-1 ❌ unit 204m 56s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 49s The patch does not generate ASF License warnings.
359m 1s
Reason Tests
Failed junit tests hadoop.hdfs.tools.TestDFSAdmin
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/4/artifact/out/Dockerfile
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname Linux 812137fde625 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ba26e23
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/4/testReport/
Max. process+thread count 4602 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@fateh288
Copy link
Contributor Author

Style error #1: -nonSuperUser is protected and is in-line with other similar member variables of the class.
Style error #2: not related to PR. Only whitespace removed from line for style check
Style error #3,#4,#5: not related to PR. Only whitespace removed from line for style check
Style error #6: Line is longer than 100 characters -- can be fixed if really necessary, else can be followed up in another future PR. @dineshchitlangia do you suggest fixing this now (somehow the dev-support/bin/test-patch does not pick up any of these style errors which the CI pipeline is throwing here in the PR)?

Unit test failures: not related to the patch

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 25s Maven dependency ordering for branch
+1 💚 mvninstall 36m 40s trunk passed
+1 💚 compile 19m 54s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 18m 3s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 checkstyle 4m 46s trunk passed
+1 💚 mvnsite 3m 26s trunk passed
+1 💚 javadoc 2m 35s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 2m 50s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 6m 8s trunk passed
+1 💚 shadedclient 41m 15s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 2m 12s the patch passed
+1 💚 compile 18m 58s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 18m 58s the patch passed
+1 💚 compile 17m 48s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 javac 17m 48s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 5m 20s /results-checkstyle-root.txt root: The patch generated 6 new + 113 unchanged - 4 fixed = 119 total (was 117)
+1 💚 mvnsite 3m 20s the patch passed
+1 💚 javadoc 2m 30s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 2m 47s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 6m 31s the patch passed
+1 💚 shadedclient 42m 7s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 31s hadoop-common in the patch passed.
-1 ❌ unit 262m 24s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 13s The patch does not generate ASF License warnings.
539m 1s
Reason Tests
Failed junit tests hadoop.hdfs.TestRollingUpgrade
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/3/artifact/out/Dockerfile
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname Linux 166ae097dfc2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ba26e23
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/3/testReport/
Max. process+thread count 3134 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@fateh288
Copy link
Contributor Author
fateh288 commented Apr 1, 2024

@dineshchitlangia
Requesting your feedback

@fateh288
Copy link
Contributor Author

Requesting review on this patch.
The style check failures are from legacy code and not introduced in this patch specifically.
The unit test failures are also unrelated (the same patch passed the unit tests previously and no logic changes done in the follow up patch - only style changes fixed)

@hadoop-yetus
Copy link

💔 -1 overall

8000
Vote Subsystem Runtime Logfile Comment
_ Prechecks _
+1 💚 dupname 0m 01s No case conflicting files found.
+0 🆗 codespell 0m 01s codespell was not available.
+0 🆗 detsecrets 0m 01s detect-secrets was not available.
+0 🆗 markdownlint 0m 01s markdownlint was not available.
+0 🆗 spotbugs 0m 01s spotbugs executables are not available.
+1 💚 @author 0m 00s The patch does not contain any @author tags.
+1 💚 test4tests 0m 00s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 2m 40s Maven dependency ordering for branch
+1 💚 mvninstall 105m 33s trunk passed
+1 💚 compile 46m 51s trunk passed
+1 💚 checkstyle 7m 01s trunk passed
-1 ❌ mvnsite 5m 22s /branch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in trunk failed.
+1 💚 javadoc 12m 20s trunk passed
+1 💚 shadedclient 192m 14s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 2m 39s Maven dependency ordering for patch
+1 💚 mvninstall 11m 28s the patch passed
+1 💚 compile 44m 13s the patch passed
+1 💚 javac 44m 13s the patch passed
+1 💚 blanks 0m 01s The patch has no blanks issues.
+1 💚 checkstyle 6m 53s the patch passed
-1 ❌ mvnsite 5m 10s /patch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
+1 💚 javadoc 12m 21s the patch passed
+1 💚 shadedclient 204m 00s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 6m 31s The patch does not generate ASF License warnings.
623m 42s
Subsystem Report/Notes
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname MINGW64_NT-10.0-17763 6776337c4aef 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool maven
Personality /c/hadoop/dev-support/bin/hadoop.sh
git revision trunk / ba26e23
Default Java Azul Systems, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6677/1/testReport/
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6677/1/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@sodonnel
Copy link
Contributor

@fateh288 This change LGTM. As the last commit was a little old, could you merge in trunk or rebase on trunk and push the change so the CI runs again. This will avoid any problems that may occur due to some change on trunk over the last couple of months. Thanks!

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 35s Maven dependency ordering for branch
+1 💚 mvninstall 37m 2s trunk passed
+1 💚 compile 19m 31s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 compile 18m 5s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 checkstyle 4m 52s trunk passed
+1 💚 mvnsite 3m 24s trunk passed
+1 💚 javadoc 2m 35s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 2m 51s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 6m 12s trunk passed
+1 💚 shadedclient 42m 14s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 33s Maven dependency ordering for patch
+1 💚 mvninstall 2m 10s the patch passed
+1 💚 compile 18m 33s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javac 18m 33s the patch passed
+1 💚 compile 17m 52s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 javac 17m 52s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 44s /results-checkstyle-root.txt root: The patch generated 6 new + 113 unchanged - 4 fixed = 119 total (was 117)
+1 💚 mvnsite 3m 21s the patch passed
+1 💚 javadoc 2m 30s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 2m 51s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 6m 28s the patch passed
+1 💚 shadedclient 42m 4s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 30s hadoop-common in the patch passed.
+1 💚 unit 263m 0s hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 15s The patch does not generate ASF License warnings.
536m 52s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/5/artifact/out/Dockerfile
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname Linux 13832eaa7101 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ec062df
Default Java Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/5/testReport/
Max. process+thread count 3134 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@sodonnel
Copy link
Contributor

@fateh288 there are still some checkstyle violations on lines that are changed in this PR:

./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java:178:    protected boolean nonSuperUser = false; // enter/exit safe mode:23: Variable 'nonSuperUser' must be private and have accessor methods. [VisibilityModifier]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java:1044:    DatanodeStorage storage; //only one storage:21: Variable 'storage' must be private and have accessor methods. [VisibilityModifier]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java:1177:    private int transferBlocks( Block blocks[],:31: '(' is followed by whitespace. [ParenPad]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java:1177:    private int transferBlocks( Block blocks[],:39: 'blocks' hides a field. [HiddenField]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java:1177:    private int transferBlocks( Block blocks[],:45: Array brackets at illegal position. [ArrayTypeStyle]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java:1298:          LOG.error("Potentially insufficient permission: try running the tool with -nonSuperUser argument or login as super user");: Line is longer than 100 characters (found 132). [LineLength]

If you could address these, we are good to commit this.

@fateh288
Copy link
Contributor Author

@sodonnel
The style checks while suggested otherwise from CI process are not really from this commit (e.g. 2,3,4,5) if we look closely or are not blockers (e.g. 1) . I have fixed number 6.

Style error 1: -nonSuperUser is protected and is in-line with other similar member variables of the class.
Style error 2: not related to PR. Only whitespace removed from line for style check
Style error 3,4,5: not related to PR. Only whitespace removed from this line for an existing style check violation which I fixed
Style error 6: Line is longer than 100 characters -- Fixed

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 45s Maven dependency ordering for branch
+1 💚 mvninstall 19m 56s trunk passed
+1 💚 compile 8m 47s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 compile 8m 6s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 checkstyle 2m 6s trunk passed
+1 💚 mvnsite 1m 55s trunk passed
+1 💚 javadoc 1m 35s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 1m 54s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 3m 23s trunk passed
+1 💚 shadedclient 21m 55s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for patch
+1 💚 mvninstall 1m 9s the patch passed
+1 💚 compile 8m 27s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javac 8m 27s the patch passed
+1 💚 compile 8m 7s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 javac 8m 7s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 1s /results-checkstyle-root.txt root: The patch generated 5 new + 113 unchanged - 4 fixed = 118 total (was 117)
+1 💚 mvnsite 1m 49s the patch passed
+1 💚 javadoc 1m 30s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 1m 53s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 3m 30s the patch passed
+1 💚 shadedclient 21m 52s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 35s hadoop-common in the patch passed.
+1 💚 unit 197m 44s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
351m 20s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/6/artifact/out/Dockerfile
GITHUB PR #6677
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname Linux b75a01bf495c 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 61e7bb1
Default Java Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/6/testReport/
Max. process+thread count 4437 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6677/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

sodonnel
sodonnel approved these changes Jun 18, 2024
View reviewed changes
Copy link
Contributor
@sodonnel sodonnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Any style violations are in existing code lines or keeping with the style of the existing code (ie protected variable rather than private).

@sodonnel sodonnel merged commit 90024d8 into apache:trunk Jun 18, 2024
4 checks passed
KeeProMise pushed a commit to KeeProMise/hadoop that referenced this pull request Sep 9, 2024
@fateh288 @KeeProMise
HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful f…
6d1f36c 
…or testing auth frameworks such as Ranger (apache#6677)
Hexiaoqiao pushed a commit to Hexiaoqiao/hadoop that referenced this pull request Sep 12, 2024
@fateh288 @Hexiaoqiao
HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful f…
c5cc42e 
…or testing auth frameworks such as Ranger (apache#6677)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sodonnel sodonnel sodonnel approved these changes

@dineshchitlangia dineshchitlangia Awaiting requested review from dineshchitlangia

Assignees
No one assigned
Labels
Common HDFS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@fateh288 @hadoop-yetus @sodonnel @dineshchitlangia
0