HDFS-16396. Reconfig slow peer parameters for datanode #3827
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Hi @jojochuang @ayushtkn @sodonnel @Hexiaoqiao @ferhui , could you please take a look at this. Thanks. |
|
🎊 +1 overall
This message was automatically generated. |
ayushtkn
reviewed
Jan 1, 2022
...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
Outdated
Show resolved
Hide resolved
| peerMetrics = DataNodePeerMetrics.create(getDisplayName(), getConf()); | ||
| } | ||
| } else { | ||
| peerMetrics = null; |
There was a problem hiding this comment.
peerMetrics isn't synchronised and it is being fetched by BpServiceActor & BlockReciever. I doubt in race conditions this may lead to some NPE there
There was a problem hiding this comment.
peerMetricsisn't synchronised and it is being fetched by BpServiceActor & BlockReciever. I doubt in race conditions this may lead to some NPE there
Thanks @ayushtkn for your comment.
To avoid NPE, I did not set peerMetrics to null when disable, but peerMetrics remains. Do you think such a solution is feasible? Thank you.
...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
Outdated
Show resolved
Hide resolved
...p-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
Outdated
Show resolved
Hide resolved
|
Hi @tamaashu , please also take a look at this. Thank you. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
tasanuma
reviewed
Feb 14, 2022
...p-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
Outdated
Show resolved
Hide resolved
Comment on lines
84
to
85
| this.slowNodeDetector = | ||
| new OutlierDetector(minOutlierDetectionNodes, lowThresholdMs); |
There was a problem hiding this comment.
this.slowNodeDetector has to update minOutlierDetectionNodes and lowThresholdMs after reconfiguring them, right?
There was a problem hiding this comment.
Thanks @tasanuma for your comment, and it makes sense to me. I updated the code, please have a look when you are free.
|
🎊 +1 overall
This message was automatically generated. |
tasanuma
pushed a commit
that referenced
this pull request
Feb 16, 2022
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> (cherry picked from commit 0c194f2)
1 task
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: HDFS-16396.
In large clusters, rolling restart datanodes takes a long time. We can make slow peers parameters and slow disks parameters in datanode reconfigurable to facilitate cluster operation and maintenance.