8000 HDDS-1881. Design doc: decommissioning in Ozone by elek · Pull Request #1196 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HDDS-1881. Design doc: decommissioning in Ozone #1196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

8000 Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 28, 2019

Conversation

elek
Copy link
Member
@elek elek commented Jul 31, 2019

Design doc can be attached to the documentation. In this jira the design doc will be attached and merged to the documentation page.

See: https://issues.apache.org/jira/browse/HDDS-1881

@elek elek added the ozone label Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019
@apache apache deleted a comment from hadoop-yetus Jul 31, 2019

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 50 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 595 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1380 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 564 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 690 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 54 The patch does not generate ASF License warnings.
2921
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/5/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux 8c5345d9063f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / a7371a7
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/5/artifact/out/whitespace-tabs.txt
Max. process+thread count 412 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/5/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 108 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 703 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1614 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 604 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 768 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 47 The patch does not generate ASF License warnings.
3317
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/6/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux 146cdd28344c 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / f86de6f
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/6/artifact/out/whitespace-tabs.txt
Max. process+thread count 314 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/6/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 44 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 656 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1439 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 543 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 633 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 38 The patch does not generate ASF License warnings.
2849
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/7/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux 87b0f790b823 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 70b4617
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/7/artifact/out/whitespace-tabs.txt
Max. process+thread count 444 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/7/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 70 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 618 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1474 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 610 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 776 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 44 The patch does not generate ASF License warnings.
3135
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/8/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux 80fc939f5cb2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 00b5a27
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/8/artifact/out/whitespace-tabs.txt
Max. process+thread count 306 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/8/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

< 9E88 input type="hidden" name="_method" value="put" autocomplete="off" />

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 48 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 621 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1349 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 542 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 673 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 40 The patch does not generate ASF License warnings.
2803
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/9/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux ac7dcc283e37 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 9a1d8cf
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/9/artifact/out/whitespace-tabs.txt
Max. process+thread count 442 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/9/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 44 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 656 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1448 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 556 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 664 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 52 The patch does not generate ASF License warnings.
2910
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/10/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux f079a366a05b 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 6244502
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/10/artifact/out/whitespace-tabs.txt
Max. process+thread count 452 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/10/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 43 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
-1 mvninstall 122 hadoop-hdds in trunk failed.
-1 mvninstall 6 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 137 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 mvninstall 7 hadoop-hdds in the patch failed.
-1 mvninstall 29 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch has 114 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-1 whitespace 1 The patch 1202 line(s) with tabs.
-1 shadedclient 29 patch has errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 23 The patch does not generate ASF License warnings.
352
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux 958ce69b648d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 69ddb36
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/branch-mvninstall-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/patch-mvninstall-hadoop-ozone.txt
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/whitespace-eol.txt
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/artifact/out/whitespace-tabs.txt
Max. process+thread count 94 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/11/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

Node state | Variable (# of containers) |
--------------------------------------|---------------------------------|
HEALTHY | `healthy` |
STALE + DEAD + DECOMMISSIONED | |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:tabs in line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 39 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 mvninstall 606 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1404 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 578 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch 2 line(s) with tabs.
+1 shadedclient 690 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 asflicense 51 The patch does not generate ASF License warnings.
2934
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/12/artifact/out/Dockerfile
GITHUB PR #1196
Optional Tests dupname asflicense mvnsite
uname Linux b9f9e34f3766 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3329257
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/12/artifact/out/whitespace-tabs.txt
Max. process+thread count 412 (vs. ulimit of 5500)
modules C: hadoop-hdds/docs U: hadoop-hdds/docs
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1196/12/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


The second kind is when the failure is 'transient.' The user knows that this failure is temporary and cluster in most cases can safely ignore this issue. However, if the transient failures are going to cause a failure of availability; then the user would like the Ozone to take appropriate actions to address it. An example of this case, is if the user put 3 data nodes into maintenance mode and switched them off.

The transient failure can violate the availability guarantees of Ozone; Since the user is telling us not to take corrective actions. Many times, the user does not understand the impact on availability while asking Ozone to ignore the failure.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -> Some

@anuengineer anuengineer merged commit c7d426d into apache:trunk Aug 28, 2019
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
* Address comments from Anu
 * In-flight calculation is removed from the base model
 * Throttling is mentioned but will be discussed in a separated document
 * Model is improved to handle the over-replication in a specific way
 (to handle Nanda's use-case)
 * Stop conditions are defined independently from the model
 * Style of the table fragments are improved
* typo fix (thx to sodonnell)
* remove white spaces from the end of the lines
* example is restored
* minor whitespace improvement to support pdf conversion
RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020
* Address comments from Anu
 * In-flight calculation is removed from the base model
 * Throttling is mentioned but will be discussed in a separated document
 * Model is improved to handle the over-replication in a specific way
 (to handle Nanda's use-case)
 * Stop conditions are defined independently from the model
 * Style of the table fragments are improved
* typo fix (thx to sodonnell)
* remove white spaces from the end of the lines
* example is restored
* minor whitespace improvement to support pdf conversion
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0