8000 HDDS-1138. Ozone Client should avoid talking to SCM directly by xiaoyuyao · Pull Request #585 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HDDS-1138. Ozone Client should avoid talking to SCM directly #585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 15, 2019
Merged

HDDS-1138. Ozone Client should avoid talking to SCM directly #585

merged 3 commits into from
Mar 15, 2019

Conversation

xiaoyuyao
Copy link
Contributor

This PR is build based on @mukul1987 's initial patch to optimize write path only.

The difference are

  • persist pipeline info into OM keylocation metadata
  • support both read and write without client talking to scm
  • There are limitations in this PR when the pipeline changes at SCM side. I will send additional code changes to support refresh pipeline info for read. (Basic idea documented in the 2nd commits).

@xiaoyuyao xiaoyuyao self-assigned this Mar 9, 2019
@xiaoyuyao xiaoyuyao added the ozone label Mar 9, 2019
@xiaoyuyao xiaoyuyao requested a review from jnp March 9, 2019 22:00
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 43 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 55 Maven dependency ordering for branch
+1 mvninstall 1053 trunk passed
+1 compile 96 trunk passed
+1 checkstyle 28 trunk passed
+1 mvnsite 123 trunk passed
+1 shadedclient 854 branch has no errors when building and testing our client artifacts.
+1 findbugs 167 trunk passed
+1 javadoc 99 trunk passed
_ Patch Compile Tests _
0 mvndep 12 Maven dependency ordering for patch
+1 mvninstall 129 the patch passed
+1 compile 87 the patch passed
+1 cc 87 the patch passed
+1 javac 87 the patch passed
-0 checkstyle 23 hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 mvnsite 97 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 759 patch has no errors when building and testing our client artifacts.
+1 findbugs 185 the patch passed
+1 javadoc 89 the patch passed
_ Other Tests _
+1 unit 37 common in the patch passed.
+1 unit 25 client in the patch passed.
+1 unit 41 ozone-manager in the patch passed.
+1 unit 25 objectstore-service in the patch passed.
+1 asflicense 26 The patch does not generate ASF License warnings.
4072
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-585/1/artifact/out/Dockerfile
GITHUB PR #585
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 828589722e44 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ddede7a
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-585/1/artifact/out/diff-checkstyle-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-585/1/testReport/
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-585/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@mukul1987
Copy link
Contributor

Thanks for working on this @xiaoyuyao.
+1, the changes looks good to me.

@xiaoyuyao
Copy link
Contributor Author

Rebase after the following two conflicting changes:
HDDS-1220. KeyManager#openKey should release the bucket lock before doing an allocateBlock. Contributed by Lokesh Jain.
HDDS-1095. OzoneManager#openKey should do multiple block allocations in a single SCM rpc call. Contributed by Mukul Kumar Singh.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 25 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 62 Maven dependency ordering for branch
+1 mvninstall 984 trunk passed
+1 compile 109 trunk passed
+1 checkstyle 24 trunk passed
+1 mvnsite 128 trunk passed
+1 shadedclient 787 branch has no errors when building and testing our client artifacts.
+1 findbugs 169 trunk passed
+1 javadoc 105 trunk passed
_ Patch Compile Tests _
0 mvndep 13 Maven dependency ordering for patch
+1 mvninstall 139 the patch passed
+1 compile 91 the patch passed
+1 cc 91 the patch passed
+1 javac 91 the patch passed
-0 checkstyle 23 hadoop-ozone: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
+1 mvnsite 105 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 742 patch has no errors when building and testing our client artifacts.
+1 findbugs 198 the patch passed
+1 javadoc 95 the patch passed
_ Other Tests _
+1 unit 36 common in the patch passed.
+1 unit 27 client in the patch passed.
+1 unit 40 ozone-manager in the patch passed.
+1 unit 27 objectstore-service in the patch passed.
+1 asflicense 29 The patch does not generate ASF License warnings.
3964
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-585/2/artifact/out/Dockerfile
GITHUB PR #585
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux fc8ac8483089 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d60673c
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-585/2/artifact/out/diff-checkstyle-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-585/2/testReport/
Max. process+thread count 445 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-585/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

A33F
@xiaoyuyao xiaoyuyao merged commit d1afa03 into apache:trunk Mar 15, 2019
asfgit pushed a commit that referenced this pull request Mar 15, 2019
@xiaoyuyao
HDDS-1138. Ozone Client should avoid talking to SCM directly. Contrib…
3ca1024 
…uted by Xiaoyu Yao and Mukul Kumar Singh.

Closes #585 

(cherry picked from commit d1afa03)
shanthoosh added a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
@shanthoosh @jagadish-v0
SAMZA-1788: Add LocationIdProvider abstraction.
7bb2061 
Currently in standalone, by default hostName of the standalone processor is used as LocationId. However, for containerized environments like azure cloud, kubernetes this defaulting does not work. Standalone processors can be launched from different kubernetes container on a physical machine(where each kubernetes container has different locatliyID than other kubernetes container within same machine).

To solve this problem, we introduce locationID abstraction to allow users to plugin a uniqueId identifying the execution environment of the processor.

In containerized environments, LocationId is a composite key of multiple fields: (sliceId, containerId, hostname) By default hostname will be used as LocationId(if not configured by the user).

All the processors of an application registered from an locationID should be able to share(read/write) their local state stores. Any custom LocationIdProvider is expected to honor this contract when generating the locationID.

This patch is part of SEP-11. Please refer to it for more details.

Author: Shanthoosh Venkataraman <spvenkat@usc.edu>

Reviewers: Jagadish<jagadish@apache.org>

Closes apache#585 from shanthoosh/add_location_id_interface
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jnp jnp Awaiting requested review from jnp

Assignees

@xiaoyuyao xiaoyuyao

Labels
ozone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xiaoyuyao @hadoop-yetus @mukul1987
0