Introduce new options for estimating table sizes #793
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a new option --estimate-table-sizes. When set, pgcopydb will use the relpages value in system catalog to estimate the size of each table. This is done by multiplying the number of pages by the page size. PostgreSQL uses a default of 8KB for page sizes, but can be changed in build time. Optionally, the user can set the environment variable PGCOPYDB_ESTIMATE_TABLE_SIZES to a boolean value to enable the option. If this option is used, we run vacuumdb --analyze-only on the source to update the relpages values before calculating the estimates. In passing, I fixed several things including the following: - Fix comment references to pg_autoctl - Remove some unused functions - Remove reference to non-existing vacuumdb command - Remove --cache and --drop-cache references that are no longer used - Fix typos in vacuum logs and comments - Remove references to pgcopydb_table_size table
dimitri
requested changes
May 29, 2024
| @@ -729,6 +729,16 @@ PGCOPYDB_SPLIT_TABLES_LARGER_THAN | |||
| When ``--split-tables-larger-than`` is ommitted from the command line, | |||
| then this environment variable is used. | |||
|
|
|||
| PGCOPYDB_ESTIMATE_TABLE_SIZES | |||
There was a problem hiding this comment.
Actual documentation coverage for the option is missing, see the option list at https://pgcopydb.readthedocs.io/en/latest/ref/pgcopydb_clone.html#options. In particular, we need to document the vacuumdb call that is implemented when using that option, as that's not transparent to the user at all.
src/bin/pgcopydb/schema.c
Outdated
Comment on lines
4888
to
4889
| int fnbytesestimate = PQfnumber(result, "bytesestimate"); | ||
| int fnbytesestimatepp = PQfnumber(result, "pg_size_pretty"); |
There was a problem hiding this comment.
Nitpicking: could we place these in the SQL query column list order?
dimitri
approved these changes
May 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a new option
--estimate-table-sizes. When set, pgcopydb will use the relpages value in system catalog to estimate the size of each table. This is done by multiplying the number of pages by the page size. PostgreSQL uses a default of 8KB for page sizes, but can be changed in build time.Optionally, the user can set the environment variable
PGCOPYDB_ESTIMATE_TABLE_SIZESto a boolean value to enable the option.If this option is used, we run
vacuumdb --analyze-onlyon the source to update the relpages values before calculating the estimates.In passing, I fixed several things including the following:
--cacheand--drop-cachereferences that are no longer usedpgcopydb_table_sizetableCloses #775