fix(pingcap/tidb): fix the cache too large problem #3037

wuhuizuo · 2024-07-30T10:45:27Z

do not cache the third party bin in prepare stage.

Signed-off-by: wuhuizuo wuhuizuo@126.com

do not cache the third party bin in prepare stage. Signed-off-by: wuhuizuo <wuhuizuo@126.com>

ti-chi-bot · 2024-07-30T10:45:34Z

I have already done a preliminary review for you, and I hope to help you do a better job.

Based on the pull request title and description, the key change in this pull request is to fix a cache problem by not caching the third-party bin in the prepare stage. Looking at the diff, there are a few changes made to the pull_br_integration_test.groovy file. The Prepare stage has been modified to remove the caching of the third-party bin and some other unnecessary cache. Instead, the third-party bin is downloaded and copied to the bin directory. Additionally, a new Prepare stage has been added before the Test stage.

One potential problem with this change is that the download of the third-party bin in the new Prepare stage is not cached. This could lead to longer build times if the download takes a long time. It might be better to cache the downloaded bin if possible.

To fix this potential problem, we can add a cache for the downloaded third-party bin in the new Prepare stage. We can modify the dir('third_party_download') block to cache the third_bin directory. Here is an example:

dir('third_party_download') {
    cache(path: "./third_bin", key: prow.getCacheKey('third-party-bin', REFS)) {
        retry(2) {
            sh label: "download third_party", script: """
                chmod +x ../tidb/br/tests/*.sh
                ${WORKSPACE}/tidb/br/tests/download_integration_test_binaries.sh master
                rm -rf bin/ && mkdir -p bin
                mv third_bin/* bin/
                ls -alh bin/
                ./bin/pd-server -V
                ./bin/tikv-server -V
                ./bin/tiflash --version
            """
        }
    }
}

With this change, the downloaded third-party bin will be cached and reused in subsequent builds if the same cache key is used.

wuhuizuo · 2024-07-30T10:50:32Z

tested in: https://do.pingcap.net/jenkins/job/pingcap/job/tidb/job/pull_br_integration_test/1368/

with just G00 enabled in matrix.

wuhuizuo · 2024-07-30T10:52:39Z

I have already done a preliminary review for you, and I hope to help you do a better job.

Based on the pull request title and description, the key change in this pull request is to fix a cache problem by not caching the third-party bin in the prepare stage. Looking at the diff, there are a few changes made to the pull_br_integration_test.groovy file. The Prepare stage has been modified to remove the caching of the third-party bin and some other unnecessary cache. Instead, the third-party bin is downloaded and copied to the bin directory. Additionally, a new Prepare stage has been added before the Test stage.

One potential problem with this change is that the download of the third-party bin in the new Prepare stage is not cached. This could lead to longer build times if the download takes a long time. It might be better to cache the downloaded bin if possible.

To fix this potential problem, we can add a cache for the downloaded third-party bin in the new Prepare stage. We can modify the dir('third_party_download') block to cache the third_bin directory. Here is an example:
dir('third_party_download') {
    cache(path: "./third_bin", key: prow.getCacheKey('third-party-bin', REFS)) {
        retry(2) {
            sh label: "download third_party", script: """
                chmod +x ../tidb/br/tests/*.sh
                ${WORKSPACE}/tidb/br/tests/download_integration_test_binaries.sh master
                rm -rf bin/ && mkdir -p bin
                mv third_bin/* bin/
                ls -alh bin/
                ./bin/pd-server -V
                ./bin/tikv-server -V
                ./bin/tiflash --version
            """
        }
    }
}
With this change, the downloaded third-party bin will be cached and reused in subsequent builds if the same cache key is used.

I known, but the issue is to workaround the cache size limit problem.

ti-chi-bot · 2024-07-30T10:53:24Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

pipelines/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fix(pingcap/tidb): fix the cache too large problem

7de4ddb

do not cache the third party bin in prepare stage. Signed-off-by: wuhuizuo <wuhuizuo@126.com>

ti-chi-bot bot added the area/jenkins-pipelines label Jul 30, 2024

ti-chi-bot bot requested a review from purelind July 30, 2024 10:45

ti-chi-bot bot added the size/M label Jul 30, 2024

wuhuizuo added lgtm approved labels Jul 30, 2024

ti-chi-bot bot merged commit 84ea7a4 into main Jul 30, 2024
2 checks passed

ti-chi-bot bot deleted the fix/br-it branch July 30, 2024 10:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(pingcap/tidb): fix the cache too large problem #3037

fix(pingcap/tidb): fix the cache too large problem #3037

fix(pingcap/tidb): fix the cache too large problem #3037

fix(pingcap/tidb): fix the cache too large problem #3037

Conversation