8000 Parquet: Benchmark Select() by jesusvazquez · Pull Request #11387 · grafana/mimir · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Parquet: Benchmark Select() #11387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 7 commits into
base: parquet-mimir
Choose a base branch
from
Draft

Parquet: Benchmark Select() #11387

wants to merge 7 commits into from

Conversation

jesusvazquez
Copy link
Member

Some notes

  • Its benchmarking against BlockQueryable but a fairer comparison would be against BlockStoreQueryable since that one also uses a bucket however did not manage to get it running because of the ring and I just wanted to get something quick going.
  • Had to create my own CreateBlock() because current codebase has no support for histograms. The custom CreateBlock only creates float series.
  • Run instructions:
MIMIR_SELECT_QUERYABLE=parquet go test -v -run=^$ -bench=^BenchmarkQueryableSelect$ -count=6 -benchmem ./pkg/querier/... > parquet_bench.txt
MIMIR_SELECT_QUERYABLE=block  go test -v -run=^$ -bench=^BenchmarkQueryableSelect$ -count=6 -benchmem ./pkg/querier/... > block_bench.txt

# Then run benchstat or benchcmp
benchstat parquet_bench.txt block_bench.txt
Benchstat 
❯ benchstat parquet_bench.txt block_bench.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/querier
cpu: Apple M2 Pro
                                                  │ parquet_bench.txt │          block_bench.txt           │
                                                  │      sec/op       │   sec/op     vs base               │
QueryableSelect/SingleMetricExact-12                   1922.694µ ± 1%   6.367µ ± 1%  -99.67% (p=0.002 n=6)
QueryableSelect/SingleMetricAllInstances-12            1846.357µ ± 1%   6.018µ ± 1%  -99.67% (p=0.002 n=6)
QueryableSelect/MultipleMetricsRange-12                 4091.92µ ± 1%   20.50µ ± 0%  -99.50% (p=0.002 n=6)
QueryableSelect/MultipleMetricsSparse-12                3749.50µ ± 1%   20.75µ ± 2%  -99.45% (p=0.002 n=6)
QueryableSelect/HighCardinalitySingleMetric-12         1917.564µ ± 3%   8.094µ ± 3%  -99.58% (p=0.002 n=6)
QueryableSelect/HighCardinalityMultipleMetrics-12      2061.119µ ± 4%   9.493µ ± 4%  -99.54% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsSingleMetric-12          131.947µ ± 1%   2.882µ ± 2%  -97.82% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsMultipleMetrics-12       330.712µ ± 0%   5.149µ ± 1%  -98.44% (p=0.002 n=6)
QueryableSelect/NegativeRegexSingleMetric-12            132.863µ ± 0%   6.283µ ± 3%  -95.27% (p=0.002 n=6)
QueryableSelect/NegativeRegexMultipleMetrics-12         2056.88µ ± 1%   14.13µ ± 1%  -99.31% (p=0.002 n=6)
QueryableSelect/NonExistentValue-12                     90427.5n ± 2%   687.0n ± 6%  -99.24% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexSingleMetric-12          1928.618µ ± 3%   8.191µ ± 5%  -99.58% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexMultipleMetrics-12        3426.10µ ± 2%   16.25µ ± 2%  -99.53% (p=0.002 n=6)
QueryableSelect/SparseSeriesSingleMetric-12             188.332µ ± 1%   2.894µ ± 1%  -98.46% (p=0.002 n=6)
QueryableSelect/SparseSeriesMultipleMetrics-12          312.372µ ± 0%   4.144µ ± 1%  -98.67% (p=0.002 n=6)
geomean                                                   851.8µ        6.564µ       -99.23%

                                                  │ parquet_bench.txt │          block_bench.txt           │
                                                  │     series/op     │ series/op   vs base                │
QueryableSelect/SingleMetricExact-12                     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SingleMetricAllInstances-12              1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/MultipleMetricsRange-12                  5.000 ± 0%     5.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/MultipleMetricsSparse-12                 5.000 ± 0%     5.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/HighCardinalitySingleMetric-12           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/HighCardinalityMultipleMetrics-12        1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SubsetSelectorsSingleMetric-12           0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SubsetSelectorsMultipleMetrics-12        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/NegativeRegexSingleMetric-12             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/NegativeRegexMultipleMetrics-12          1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/NonExistentValue-12                      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/ExpensiveRegexSingleMetric-12            1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/ExpensiveRegexMultipleMetrics-12         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SparseSeriesSingleMetric-12              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SparseSeriesMultipleMetrics-12           0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                             ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                  │ parquet_bench.txt │           block_bench.txt           │
                                                  │       B/op        │     B/op      vs base               │
QueryableSelect/SingleMetricExact-12                  10138.71Ki ± 6%   17.26Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/SingleMetricAllInstances-12           10096.31Ki ± 2%   17.13Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/MultipleMetricsRange-12               33422.87Ki ± 2%   32.91Ki ± 0%  -99.90% (p=0.002 n=6)
QueryableSelect/MultipleMetricsSparse-12              28799.35Ki ± 1%   32.91Ki ± 0%  -99.89% (p=0.002 n=6)
QueryableSelect/HighCardinalitySingleMetric-12        10202.93Ki ± 3%   17.27Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/HighCardinalityMultipleMetrics-12     10208.42Ki ± 5%   17.66Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsSingleMetric-12          202786.5 ± 0%    1008.0 ± 0%  -99.50% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsMultipleMetrics-12      135.338Ki ± 0%   1.625Ki ± 0%  -98.80% (p=0.002 n=6)
QueryableSelect/NegativeRegexSingleMetric-12           202.643Ki ± 0%   4.062Ki ± 0%  -98.00% (p=0.002 n=6)
QueryableSelect/NegativeRegexMultipleMetrics-12        9951.51Ki ± 3%   20.86Ki ± 0%  -99.79% (p=0.002 n=6)
QueryableSelect/NonExistentValue-12                     190556.0 ± 0%     813.0 ± 0%  -99.57% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexSingleMetric-12         10205.23Ki ± 1%   17.91Ki ± 0%  -99.82% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexMultipleMetrics-12      22914.83Ki ± 2%   25.87Ki ± 0%  -99.89% (p=0.002 n=6)
QueryableSelect/SparseSeriesSingleMetric-12             220247.0 ± 0%    1008.0 ± 0%  -99.54% (p=0.002 n=6)
QueryableSelect/SparseSeriesMultipleMetrics-12         138.391Ki ± 1%   1.375Ki ± 0%  -99.01% (p=0.002 n=6)
geomean                                                  2.400Mi        7.155Ki       -99.71%

                                                  │ parquet_bench.txt │          block_bench.txt          │
                                                  │     allocs/op     │ allocs/op   vs base               │
QueryableSelect/SingleMetricExact-12                     2161.00 ± 0%   40.00 ± 0%  -98.15% (p=0.002 n=6)
QueryableSelect/SingleMetricAllInstances-12              2021.50 ± 0%   35.00 ± 0%  -98.27% (p=0.002 n=6)
QueryableSelect/MultipleMetricsRange-12                  2554.00 ± 0%   86.00 ± 0%  -96.63% (p=0.002 n=6)
QueryableSelect/MultipleMetricsSparse-12                 2521.00 ± 0%   89.00 ± 0%  -96.47% (p=0.002 n=6)
QueryableSelect/HighCardinalitySingleMetric-12           2162.00 ± 0%   40.00 ± 0%  -98.15% (p=0.002 n=6)
QueryableSelect/HighCardinalityMultipleMetrics-12        2243.00 ± 0%   45.00 ± 0%  -97.99% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsSingleMetric-12            340.00 ± 0%   20.00 ± 0%  -94.12% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsMultipleMetrics-12         547.00 ± 0%   27.00 ± 0%  -95.06% (p=0.002 n=6)
QueryableSelect/NegativeRegexSingleMetric-12              327.00 ± 0%   45.00 ± 0%  -86.24% (p=0.002 n=6)
QueryableSelect/NegativeRegexMultipleMetrics-12          2236.50 ± 0%   74.00 ± 0%  -96.69% (p=0.002 n=6)
QueryableSelect/NonExistentValue-12                       276.00 ± 0%   15.00 ± 0%  -94.57% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexSingleMetric-12            2164.50 ± 0%   47.00 ± 0%  -97.83% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexMultipleMetrics-12         2497.00 ± 0%   74.00 ± 0%  -97.04% (p=0.002 n=6)
QueryableSelect/SparseSeriesSingleMetric-12               465.00 ± 0%   20.00 ± 0%  -95.70% (p=0.002 n=6)
QueryableSelect/SparseSeriesMultipleMetrics-12            548.00 ± 0%   25.00 ± 0%  -95.44% (p=0.002 n=6)
geomean                                                   1.139k        39.55       -96.53%
Benccmp 
❯ benchstat parquet_bench.txt block_bench.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/querier
cpu: Apple M2 Pro
                                                  │ parquet_bench.txt │          block_bench.txt           │
                                                  │      sec/op       │   sec/op     vs base               │
QueryableSelect/SingleMetricExact-12                   1922.694µ ± 1%   6.367µ ± 1%  -99.67% (p=0.002 n=6)
QueryableSelect/SingleMetricAllInstances-12            1846.357µ ± 1%   6.018µ ± 1%  -99.67% (p=0.002 n=6)
QueryableSelect/MultipleMetricsRange-12                 4091.92µ ± 1%   20.50µ ± 0%  -99.50% (p=0.002 n=6)
QueryableSelect/MultipleMetricsSparse-12                3749.50µ ± 1%   20.75µ ± 2%  -99.45% (p=0.002 n=6)
QueryableSelect/HighCardinalitySingleMetric-12         1917.564µ ± 3%   8.094µ ± 3%  -99.58% (p=0.002 n=6)
QueryableSelect/HighCardinalityMultipleMetrics-12      2061.119µ ± 4%   9.493µ ± 4%  -99.54% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsSingleMetric-12          131.947µ ± 1%   2.882µ ± 2%  -97.82% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsMultipleMetrics-12       330.712µ ± 0%   5.149µ ± 1%  -98.44% (p=0.002 n=6)
QueryableSelect/NegativeRegexSingleMetric-12            132.863µ ± 0%   6.283µ ± 3%  -95.27% (p=0.002 n=6)
QueryableSelect/NegativeRegexMultipleMetrics-12         2056.88µ ± 1%   14.13µ ± 1%  -99.31% (p=0.002 n=6)
QueryableSelect/NonExistentValue-12                     90427.5n ± 2%   687.0n ± 6%  -99.24% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexSingleMetric-12          1928.618µ ± 3%   8.191µ ± 5%  -99.58% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexMultipleMetrics-12        3426.10µ ± 2%   16.25µ ± 2%  -99.53% (p=0.002 n=6)
QueryableSelect/SparseSeriesSingleMetric-12             188.332µ ± 1%   2.894µ ± 1%  -98.46% (p=0.002 n=6)
QueryableSelect/SparseSeriesMultipleMetrics-12          312.372µ ± 0%   4.144µ ± 1%  -98.67% (p=0.002 n=6)
geomean                                                   851.8µ        6.564µ       -99.23%

                                                  │ parquet_bench.txt │          block_bench.txt           │
                                                  │     series/op     │ series/op   vs base                │
QueryableSelect/SingleMetricExact-12                     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SingleMetricAllInstances-12              1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/MultipleMetricsRange-12                  5.000 ± 0%     5.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/MultipleMetricsSparse-12                 5.000 ± 0%     5.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/HighCardinalitySingleMetric-12           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/HighCardinalityMultipleMetrics-12        1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SubsetSelectorsSingleMetric-12           0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SubsetSelectorsMultipleMetrics-12        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/NegativeRegexSingleMetric-12             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/NegativeRegexMultipleMetrics-12          1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/NonExistentValue-12                      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/ExpensiveRegexSingleMetric-12            1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/ExpensiveRegexMultipleMetrics-12         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SparseSeriesSingleMetric-12              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
QueryableSelect/SparseSeriesMultipleMetrics-12           0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                             ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                  │ parquet_bench.txt │           block_bench.txt           │
                                                  │       B/op        │     B/op      vs base               │
QueryableSelect/SingleMetricExact-12                  10138.71Ki ± 6%   17.26Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/SingleMetricAllInstances-12           10096.31Ki ± 2%   17.13Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/MultipleMetricsRange-12               33422.87Ki ± 2%   32.91Ki ± 0%  -99.90% (p=0.002 n=6)
QueryableSelect/MultipleMetricsSparse-12              28799.35Ki ± 1%   32.91Ki ± 0%  -99.89% (p=0.002 n=6)
QueryableSelect/HighCardinalitySingleMetric-12        10202.93Ki ± 3%   17.27Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/HighCardinalityMultipleMetrics-12     10208.42Ki ± 5%   17.66Ki ± 0%  -99.83% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsSingleMetric-12          202786.5 ± 0%    1008.0 ± 0%  -99.50% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsMultipleMetrics-12      135.338Ki ± 0%   1.625Ki ± 0%  -98.80% (p=0.002 n=6)
QueryableSelect/NegativeRegexSingleMetric-12           202.643Ki ± 0%   4.062Ki ± 0%  -98.00% (p=0.002 n=6)
QueryableSelect/NegativeRegexMultipleMetrics-12        9951.51Ki ± 3%   20.86Ki ± 0%  -99.79% (p=0.002 n=6)
QueryableSelect/NonExistentValue-12                     190556.0 ± 0%     813.0 ± 0%  -99.57% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexSingleMetric-12         10205.23Ki ± 1%   17.91Ki ± 0%  -99.82% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexMultipleMetrics-12      22914.83Ki ± 2%   25.87Ki ± 0%  -99.89% (p=0.002 n=6)
QueryableSelect/SparseSeriesSingleMetric-12             220247.0 ± 0%    1008.0 ± 0%  -99.54% (p=0.002 n=6)
QueryableSelect/SparseSeriesMultipleMetrics-12         138.391Ki ± 1%   1.375Ki ± 0%  -99.01% (p=0.002 n=6)
geomean                                                  2.400Mi        7.155Ki       -99.71%

                                                  │ parquet_bench.txt │          block_bench.txt          │
                                                  │     allocs/op     │ allocs/op   vs base               │
QueryableSelect/SingleMetricExact-12                     2161.00 ± 0%   40.00 ± 0%  -98.15% (p=0.002 n=6)
QueryableSelect/SingleMetricAllInstances-12              2021.50 ± 0%   35.00 ± 0%  -98.27% (p=0.002 n=6)
QueryableSelect/MultipleMetricsRange-12                  2554.00 ± 0%   86.00 ± 0%  -96.63% (p=0.002 n=6)
QueryableSelect/MultipleMetricsSparse-12                 2521.00 ± 0%   89.00 ± 0%  -96.47% (p=0.002 n=6)
QueryableSelect/HighCardinalitySingleMetric-12           2162.00 ± 0%   40.00 ± 0%  -98.15% (p=0.002 n=6)
QueryableSelect/HighCardinalityMultipleMetrics-12        2243.00 ± 0%   45.00 ± 0%  -97.99% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsSingleMetric-12            340.00 ± 0%   20.00 ± 0%  -94.12% (p=0.002 n=6)
QueryableSelect/SubsetSelectorsMultipleMetrics-12         547.00 ± 0%   27.00 ± 0%  -95.06% (p=0.002 n=6)
QueryableSelect/NegativeRegexSingleMetric-12              327.00 ± 0%   45.00 ± 0%  -86.24% (p=0.002 n=6)
QueryableSelect/NegativeRegexMultipleMetrics-12          2236.50 ± 0%   74.00 ± 0%  -96.69% (p=0.002 n=6)
QueryableSelect/NonExistentValue-12                       276.00 ± 0%   15.00 ± 0%  -94.57% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexSingleMetric-12            2164.50 ± 0%   47.00 ± 0%  -97.83% (p=0.002 n=6)
QueryableSelect/ExpensiveRegexMultipleMetrics-12         2497.00 ± 0%   74.00 ± 0%  -97.04% (p=0.002 n=6)
QueryableSelect/SparseSeriesSingleMetric-12               465.00 ± 0%   20.00 ± 0%  -95.70% (p=0.002 n=6)
QueryableSelect/SparseSeriesMultipleMetrics-12            548.00 ± 0%   25.00 ± 0%  -95.44% (p=0.002 n=6)
geomean                                                   1.139k        39.55       -96.53%

@jesusvazquez
querier-parquet: Add benchmark
e716247
@jesusvazquez
update
e0b2f90
@jesusvazquez
update structure
20a0271 
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
@jesusvazquez
extract cases
1133bf9
@jesusvazquez
do not add the samples
cd14e81 
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
@jesusvazquez jesusvazquez force-pushed the jvp/work-on-benchmarks branch from c45461c to cd14e81 Compare May 5, 2025 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jesusvazquez
0