Fix memory_usage when calculating nested list column
#16193
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
| @@ -926,3 +927,29 @@ def test_list_iterate_error(): | |||
| def test_list_struct_list_memory_usage(): | |||
| df = cudf.DataFrame({"a": [[{"b": [1]}]]}) | |||
| assert df.memory_usage().sum() == 16 | |||
|
|
|||
|
|
|||
| def test_empty_nested_list_uninitialized_offsets_memory_usage(): | |||
There was a problem hiding this comment.
Would it be possible to re-write this test using a public API rather than accessing column constructors?
There was a problem hiding this comment.
I tried finding a public API (constructor) that would create an empty offset column, but I think the cudf Python layer is good at consistently ensuring the offset column has at least one entry.
There are operations that go through libcudf that will return an empty offset column (e.g. .iloc[0:0]), but I wanted to ensure we test something that has the least chance to change if a public API implementation changes.
galipremsagar
approved these changes
Jul 4, 2024
wence-
approved these changes
Jul 4, 2024
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The offset column of a nested empty list column may be empty as discussed in #16164.
ListColumn.memory_usageassumed that this column was non-emptyUnblocks rapidsai/cuspatial#1400
Checklist