8000 Backport Hadoop 17215 branch 2.10 by arjun4084346 · Pull Request #4261 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Backport Hadoop 17215 branch 2.10 #4261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: branch-2.10
Choose a base branch
from
Open

Backport Hadoop 17215 branch 2.10 #4261

wants to merge 7 commits into from

Conversation

arjun4084346
Copy link
@arjun4084346 arjun4084346 commented May 3, 2022 •
edited
Loading

Description of PR

The reason for this cherry pick is to backport fix for https://issues.apache.org/jira/browse/HADOOP-17215 .The following commits were che 8000 rry picked in order to do it cleanly

8031c66
459eb2a
7f486f0
3472c3e
0d85515
e31a636
07b7d07

The second last commit e31a636 was the intended one and the last one was cherry picked because it fixed some test failures.

How was this patch tested?

Ran mvn test -pl hadoop-tools/hadoop-azure

[INFO] Results:
[INFO] 
[WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 4
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:09 min
[INFO] Finished at: 2022-05-03T09:43:05-07:00
[INFO] ------------------------------------------------------------------------

Ran all integration abfs tests using mvn -T 1C -Dparallel-tests=abfs clean verify with my storage account arjundev.dfs.core.windows.net
Storage account's Primary location: East US, Secondary location: West US

[INFO] Results:
[INFO] 
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemCreate.testFilterFSWriteAfterClose:210 » IO java.io.Fi...
[ERROR]   ITestAzureBlobFileSystemCreate.testNegativeScenariosForCreateOverwriteDisabled:363 » TokenAccessProvider
[ERROR]   ITestAzureBlobFileSystemE2E.testFlushWithFileNotFoundException:224 » IO java.i...
[ERROR]   ITestAzureBlobFileSystemE2E.testWriteWithFileNotFoundException:204 » IO java.i...
[INFO] 
[ERROR] Tests run: 426, Failures: 0, Errors: 4, Skipped: 253

[INFO] Results:
[INFO] 
[WARNING] Tests run: 151, Failures: 0, Errors: 0, Skipped: 24

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

mukund-thakur and others added 7 commits May 3, 2022 09:16
@mukund-thakur @arjun4084346
HADOOP-16965. Refactor abfs stream configuration. (apache#1956)
b03b211 
Contributed by Mukund Thakur.

(cherry picked from commit 8031c66)
@mehakmeet @arjun4084346
HADOOP-16914 Adding Output Stream Counters in ABFS (apache#1899)
957e75d 
Contributed by Mehakmeet Singh.There

(cherry picked from commit 459eb2a)
@arjun4084346
HADOOP-17016. Adding Common Counters in ABFS (apache#1991).
6fa99fa 
Contributed by: Mehakmeet Singh.

Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42
(cherry picked from commit 7f486f0)
@mehakmeet @arjun4084346
HADOOP-17065. Add Network Counters to ABFS (apache#2056)
0ca5415 
Contributed by Mehakmeet Singh.

(cherry picked from commit 3472c3e)
@mehakmeet @arjun4084346
HADOOP-17229. No updation of bytes received counter value after respo…
75a398d 
…nse failure occurs in ABFS (apache#2264)

Contributed by Mehakmeet Singh

(cherry picked from commit 0d85515)
@snvijaya @arjun4084346
HADOOP-17215: Support for conditional overwrite.
7843368 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140

(cherry picked from commit e31a636)
@steveloughran @arjun4084346
HADOOP-17325. WASB Test Failures
2280569 
Contributed by Ayush Saxena and Steve Loughran

Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
(cherry picked from commit 07b7d07)
@arjun4084346 arjun4084346 mentioned this pull request May 3, 2022
Open
4 tasks
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 11m 54s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 11 new or modified test files.
_ branch-2.10 Compile Tests _
+0 🆗 mvndep 3m 51s Maven dependency ordering for branch
+1 💚 mvninstall 14m 23s branch-2.10 passed
+1 💚 compile 14m 5s branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚 compile 11m 34s branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚 checkstyle 2m 25s branch-2.10 passed
+1 💚 mvnsite 2m 31s branch-2.10 passed
+1 💚 javadoc 2m 25s branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚 javadoc 2m 2s branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌ spotbugs 2m 42s /branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
-0 ⚠️ patch 1m 37s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 1m 20s the patch passed
+1 💚 compile 13m 18s the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚 javac 13m 18s the patch passed
+1 💚 compile 11m 39s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚 javac 11m 39s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 21s /results-checkstyle-root.txt root: The patch generated 6 new + 8 unchanged - 0 fixed = 14 total (was 8)
+1 💚 mvnsite 2m 27s the patch passed
-1 ❌ javadoc 0m 50s /results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10.txt hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems, Inc.-1.7.0_262-b10 generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚 javadoc 2m 2s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚 spotbugs 3m 55s the patch passed
_ Other Tests _
+1 💚 unit 16m 58s hadoop-common in the patch passed.
-1 ❌ unit 1m 46s /patch-unit-hadoop-tools_hadoop-azure.txt hadoop-azure in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
135m 20s
Reason Tests
Failed junit tests hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
hadoop.fs.azure.TestNativeAzureFileSystemMocked
hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
hadoop.fs.azure.TestOutOfBandAzureBlobOperations
hadoop.fs.azure.TestWasbFsck
hadoop.fs.azure.TestBlobMetadata
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/1/artifact/out/Dockerfile
GITHUB PR #4261
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 9cbed676f5a6 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-2.10 / 7843368
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/1/testReport/
Max. process+thread count 1349 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 14 new or modified test files.
_ branch-2.10 Compile Tests _
+0 🆗 mvndep 4m 3s Maven dependency ordering for branch
+1 💚 mvninstall 14m 0s branch-2.10 passed
+1 💚 compile 13m 30s branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚 compile 11m 23s branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚 checkstyle 2m 31s branch-2.10 passed
+1 💚 mvnsite 2m 58s branch-2.10 passed
+1 💚 javadoc 2m 45s branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚 javadoc 2m 33s branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌ spotbugs 2m 50s /branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
-0 ⚠️ patch 1m 55s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for patch
+1 💚 mvninstall 1m 26s the patch passed
+1 💚 compile 12m 33s the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚 javac 12m 33s the patch passed
+1 💚 compile 11m 28s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚 javac 11m 28s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 25s /results-checkstyle-root.txt root: The patch generated 6 new + 25 unchanged - 0 fixed = 31 total (was 25)
+1 💚 mvnsite 2m 54s the patch passed
-1 ❌ javadoc 1m 3s /results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10.txt hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems, Inc.-1.7.0_262-b10 generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚 javadoc 2m 30s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚 spotbugs 4m 16s the patch passed
_ Other Tests _
-1 ❌ unit 9m 58s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch passed.
+1 💚 unit 2m 1s hadoop-azure in the patch passed.
+1 💚 asflicense 1m 17s The patch does not generate ASF License warnings.
132m 56s
Reason Tests
Failed junit tests hadoop.util.TestReadWriteDiskValidator
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/2/artifact/out/Dockerfile
GITHUB PR #4261
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux c575f4115eec 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-2.10 / 2280569
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/2/testReport/
Max. process+thread count 1367 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@arjun4084346 arjun4084346 changed the title Hadoop 17215 branch 2.10 Backport Hadoop 17215 branch 2.10 May 11, 2022
@arjun4084346
Copy link
Author

@steveloughran please review/merge this. Created this one PR to cherry pick seven commits as discussed in #4171

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@arjun4084346 @hadoop-yetus @mukund-thakur @mehakmeet @snvijaya @steveloughran
0