[Feature] Change bucket number from physical partition level to materialized index level #59441
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
be/src/exec/tablet_info.h
Outdated
| @@ -258,7 +257,7 @@ class OlapTablePartitionParam { | |||
| // `invalid_row_index` stores index that chunk[index] | |||
| // has been filtered out for not being able to find tablet. | |||
| // it could be any row, becauset it's just for outputing error message for user to diagnose. | |||
| Status find_tablets(Chunk* chunk, std::vector<OlapTablePartition*>* partitions, std::vector<uint32_t>* indexes, | |||
| Status find_tablets(Chunk* chunk, std::vector<uint32_t>* hashes, std::vector<OlapTablePartition*>* partitions, | |||
There was a problem hiding this comment.
I suggest not changing the order of the parameters in this PR. It introduces an extra burden to track what you are doing.
To keep it a renaming change makes life easier.
There was a problem hiding this comment.
I update the code to keep the order of the parameters in the function find_tablets.
But I still refactor the parameters in the private function _find_tablets_with_range_partition and _find_tablets_with_list_partition. Because their original parameters cannot tell which are input and which are output, and even the original function comments are written incorrectly, some input parameters are incorrectly annotated as output parameters.
|
…ialized index level Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com>
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]❌ fail : 18 / 41 (43.90%) file detail
|
wyb
approved these changes
Jun 3, 2025
alvin-celerdata
approved these changes
Jun 3, 2025
1 task
AntiTopQuark
pushed a commit
to AntiTopQuark/starrocks
that referenced
this pull request
Jun 19, 2025
…ialized index level (StarRocks#59441) Why I'm doing: This is a preliminary work of tablet splitting and merging. Previously, bucket number is at physical partition level. All materialized indexes in a physical partition must have the same bucket number. But after tablet splitting, different materialized index in a physical partition could have different bucket number. We need to change bucket number from physical partition level to materialized index level. What I'm doing: Change bucket number from physical partition level to materialized index level. Because different materialized index in a physical partition could have different bucket number. Tablet sink cannot calculate a unified tablet index for each record of data to be distributed to different materialized index. To solve the problem, we refactor the code of tablet sink. Now tablet sink calculate a unified hash value for each record of data to be distributed to different materialized index, then the tablet index for each record of data will be calculated according to the hash value and the tablet size of a materialized index when the record of data is distributed to the materialized index. This pr just refactor the code. Next pr will remove num_bucket in OlapTablePartition and use tablets.size() in each OlapTableIndexTablets instead. Fixes StarRocks#59134 Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com> Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
This is a preliminary work of tablet splitting and merging.
Previously, bucket number is at physical partition level. All materialized indexes in a physical partition must have the same bucket number. But after tablet splitting, different materialized index in a physical partition could have different bucket number. We need to change bucket number from physical partition level to materialized index level.
What I'm doing:
Change bucket number from physical partition level to materialized index level.
Because different materialized index in a physical partition could have different bucket number. Tablet sink cannot calculate a unified tablet index for each record of data to be distributed to different materialized index.
To solve the problem, we refactor the code of tablet sink. Now tablet sink calculate a unified hash value for each record of data to be distributed to different materialized index, then the tablet index for each record of data will be calculated according to the hash value and the tablet size of a materialized index when the record of data is distributed to the materialized index.
This pr just refactor the code. Next pr will remove
num_bucketinOlapTablePartitionand usetablets.size()in eachOlapTableIndexTabletsinstead.Fixes #59134
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: