8000 Automatically make chunks immutable once all Insert operators finish by dey4ss · Pull Request #2609 · hyrise/hyrise · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Automatically make chunks immutable once all Insert operators finish #2609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 28 commits into from
Jan 31, 2024
Merged

Automatically make chunks immutable once all Insert operators finish #2609

merged 28 commits into from
Jan 31, 2024

Conversation

dey4ss
Copy link
Member
@dey4ss dey4ss commented Aug 22, 2023 •
edited
Loading

This PR automatically finalizes chunks in tables if they are filled up to the desired target size and all inserting transactions are either committed or rolled back. Though not yet part of this PR, this could trigger background statistics generation or data encoding in the future.

In the light of the discussions, we decided to rename Chunk::finalize() to set_immutable() (and added try_set_immutable()) to make the semantics clear. While the first one always marks a chunk as immutable, the second only does this if the chunk was marked to have reached its target size (done by the Insert operator that notices that and adds a new chunk) and all Insert operators writing to the chunk finished.

Alongside, this PR makes slight changes to meta tables. First, we do not build TableStatistics (histograms and NULL value ratio) for them whenever we create them anymore. Second, the ChangeMetaTable LQP node/operator was adapted to only require a second input for updates.

Furthermore, the third_party/tpcds-kit submodule is updated (again) after it was accidentally downgraded by #2618.

@dey4ss dey4ss added the FullCI Run all CI tests (slow, but required for merge) label Aug 22, 2023
@dey4ss dey4ss requested a review from Bouncner August 22, 2023 18:25
@dey4ss dey4ss changed the title Automatically finalize chunks in data tables Automatically finalize chunks once all Insert operators finish Aug 22, 2023
Bouncner
Bouncner reviewed Oct 10, 2023
View reviewed changes
Copy link
Collaborator
@Bouncner Bouncner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First batch.

@dey4ss dey4ss requested a review from Bouncner January 11, 2024 12:24
@dey4ss
Copy link
Member Author
dey4ss commented Jan 17, 2024 •
edited
Loading

@Bouncner as discussed offline, I renamed finalize() to set_immutable(), try_finalize() to try_set_immutable(), and mark_as_finalizable() to reached_target_size() to make clear what the semantics of the methods are

@dey4ss dey4ss changed the title Automatically finalize chunks once all Insert operators finish Automatically make chunks immutable once all Insert operators finish Jan 17, 2024
Bouncner
Bouncner reviewed Jan 17, 2024
View reviewed changes
@dey4ss dey4ss requested a review from Bouncner January 18, 2024 08:21
@Bouncner
Copy link
Collaborator
Bouncner commented Jan 18, 2024 •
edited
Loading

Can you add a note to the description what changed to the change tables operators?

And the changes to finalize() etc.

@dey4ss
Copy link
Member Author
dey4ss commented Jan 18, 2024

Can you add a note to the description what changed to the change tables operators?

And the changes to finalize() etc.

Anything missing?

Bouncner
Bouncner reviewed Jan 19, 2024
View reviewed changes
Bouncner
Bouncner approved these changes Jan 19, 2024
View reviewed changes
Copy link
Collaborator
@Bouncner Bouncner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few comments and one renaming discussion. Nothing that would block this PR.

Bouncner
Bouncner reviewed Jan 23, 2024
View reviewed changes
Bouncner
Bouncner reviewed Jan 23, 2024
View reviewed changes
@dey4ss dey4ss merged commit 0fed3c8 into master Jan 31, 2024
@dey4ss dey4ss deleted the dey4ss/auto-finalize branch January 31, 2024 18:53
This was referenced Mar 2, 2024
Closed
Open
This was referenced May 20, 2025
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Bouncner Bouncner Bouncner approved these changes

Assignees
No one assigned
Labels
FullCI Run all CI tests (slow, but required for merge)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dey4ss @Bouncner
0