feat(blocksync): sort peers by download rate & multiple requests for closer blocks (backport #2475) #2576
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…closer blocks (#2475) It can be reviewed commit by commit. 1. Request the block N from peer B immediately after getting `NoBlockResponse` from peer A a718fb4 2. Sort peers by download rate (the fastest peer is picked first) cf19851 3. Request a block from peer B if we are approaching pool's height (less than 50 blocks) and the current peer A is slow in sending us the block e06ebfe e97007f Closes #2379 ### Benchmarks Osmosis: without this PR (baseline): 11.45m with this PR: 7.29m ~ 36% decrease --- #### PR checklist - [x] Tests written/updated - [x] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [x] Updated relevant documentation (`docs/` or `spec/`) and code comments - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec (cherry picked from commit f8366fc) # Conflicts: # blocksync/pool.go
|
Cherry-pick of f8366fc has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
andynog
approved these changes
Mar 11, 2024
3 tasks
roy-dydx
pushed a commit
to dydxprotocol/cometbft
that referenced
this pull request
Feb 3, 2025
…closer blocks (backport cometbft#2475) (cometbft#2576) This is an automatic backport of pull request cometbft#2475 done by [Mergify](https://mergify.com). Cherry-pick of f8366fc has failed: ``` On branch mergify/bp/v0.38.x/pr-2475 Your branch is up to date with 'origin/v0.38.x'. You are currently cherry-picking commit f8366fc. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: new file: .changelog/unreleased/improvements/2475-blocksync-2nd-request.md new file: .changelog/unreleased/improvements/2475-blocksync-no-block-response.md new file: .changelog/unreleased/improvements/2475-blocksync-sort-peers.md modified: blocksync/reactor.go Unmerged paths: (use "git add <file>..." to mark resolution) both modified: blocksync/pool.go ``` To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally --- <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details> --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
roy-dydx
pushed a commit
to dydxprotocol/cometbft
that referenced
this pull request
Feb 3, 2025
…closer blocks (backport cometbft#2475) (cometbft#2576) This is an automatic backport of pull request cometbft#2475 done by [Mergify](https://mergify.com). Cherry-pick of f8366fc has failed: ``` On branch mergify/bp/v0.38.x/pr-2475 Your branch is up to date with 'origin/v0.38.x'. You are currently cherry-picking commit f8366fc. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: new file: .changelog/unreleased/improvements/2475-blocksync-2nd-request.md new file: .changelog/unreleased/improvements/2475-blocksync-no-block-response.md new file: .changelog/unreleased/improvements/2475-blocksync-sort-peers.md modified: blocksync/reactor.go Unmerged paths: (use "git add <file>..." to mark resolution) both modified: blocksync/pool.go ``` To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally --- <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details> --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
roy-dydx
pushed a commit
to dydxprotocol/cometbft
that referenced
this pull request
Feb 3, 2025
…closer blocks (backport cometbft#2475) (cometbft#2576) This is an automatic backport of pull request cometbft#2475 done by [Mergify](https://mergify.com). Cherry-pick of f8366fc has failed: ``` On branch mergify/bp/v0.38.x/pr-2475 Your branch is up to date with 'origin/v0.38.x'. You are currently cherry-picking commit f8366fc. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: new file: .changelog/unreleased/improvements/2475-blocksync-2nd-request.md new file: .changelog/unreleased/improvements/2475-blocksync-no-block-response.md new file: .changelog/unreleased/improvements/2475-blocksync-sort-peers.md modified: blocksync/reactor.go Unmerged paths: (use "git add <file>..." to mark resolution) both modified: blocksync/pool.go ``` To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally --- <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details> --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
roy-dydx
pushed a commit
to dydxprotocol/cometbft
that referenced
this pull request
Feb 3, 2025
…closer blocks (backport cometbft#2475) (cometbft#2576) This is an automatic backport of pull request cometbft#2475 done by [Mergify](https://mergify.com). Cherry-pick of f8366fc has failed: ``` On branch mergify/bp/v0.38.x/pr-2475 Your branch is up to date with 'origin/v0.38.x'. You are currently cherry-picking commit f8366fc. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: new file: .changelog/unreleased/improvements/2475-blocksync-2nd-request.md new file: .changelog/unreleased/improvements/2475-blocksync-no-block-response.md new file: .changelog/unreleased/improvements/2475-blocksync-sort-peers.md modified: blocksync/reactor.go Unmerged paths: (use "git add <file>..." to mark resolution) both modified: blocksync/pool.go ``` To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally --- <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details> --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
roy-dydx
added a commit
to dydxprotocol/cometbft
that referenced
this pull request
Feb 3, 2025
* feat(blocksync): set the max number of (concurrently) downloaded blocks (backport cometbft#2467) (cometbft#2515) Manual backport of cometbft#2467 * feat(blocksync): sort peers by download rate & multiple requests for closer blocks (backport cometbft#2475) (cometbft#2576) This is an automatic backport of pull request cometbft#2475 done by [Mergify](https://mergify.com). Cherry-pick of f8366fc has failed: ``` On branch mergify/bp/v0.38.x/pr-2475 Your branch is up to date with 'origin/v0.38.x'. You are currently cherry-picking commit f8366fc. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: new file: .changelog/unreleased/improvements/2475-blocksync-2nd-request.md new file: .changelog/unreleased/improvements/2475-blocksync-no-block-response.md new file: .changelog/unreleased/improvements/2475-blocksync-sort-peers.md modified: blocksync/reactor.go Unmerged paths: (use "git add <file>..." to mark resolution) both modified: blocksync/pool.go ``` To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally --- <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details> --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> * fix(blocksync): use timer instead of time.After (backport cometbft#2584) (cometbft#2587) This is an automatic backport of pull request cometbft#2584 done by [Mergify](https://mergify.com). --- <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details> Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> * ABC-0013 fix and broken test * blocksync pool ban test * Simplified blocksync malicious node test, fix nil pointer error * Test assertions have more detail * Update blocksync/pool_test.go Co-authored-by: Sergio Mena <sergio@informal.systems> * Update blocksync/pool_test.go Co-authored-by: Sergio Mena <sergio@informal.systems> * Update blocksync/pool_test.go * Remove one thread to make test more compact * Removed defers from test * Readded good peer to test * Update blocksync/pool_test.go * Release/v0.38.8 (cometbft#3350) <!-- Please add a reference to the issue that this PR addresses and indicate which files are most critical to review. If it fully addresses a particular issue, please include "Closes #XXX" (where "XXX" is the issue number). If this PR is non-trivial/large/complex, please ensure that you have either created an issue that the team's had a chance to respond to, or had some discussion with the team prior to submitting substantial pull requests. The team can be reached via GitHub Discussions or the Cosmos Network Discord server in the #cometbft channel. GitHub Discussions is preferred over Discord as it allows us to keep track of conversations topically. https://github.com/cometbft/cometbft/discussions If the work in this PR is not aligned with the team's current priorities, please be advised that it may take some time before it is merged - especially if it has not yet been discussed with the team. See the project board for the team's current priorities: https://github.com/orgs/cometbft/projects/1 --> Release v0.38.8 [CHANGELOG](https://github.com/cometbft/cometbft/blob/6814b5fff269f5ec6988a4832f24f4804d705ca9/CHANGELOG.md) --- - [ ] Tests written/updated - [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog) - [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments - [ ] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec * fix(test): fix TestBlockPoolMaliciousNode DATA RACE (backport cometbft#4636) (cometbft#4641) Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> * fix(test): `TestBlockPoolMaliciousNode` shutdown threads at exit (backport cometbft#4633) (cometbft#4635) This is a drive-by fix of a test that doesn't shut its threads down until the whole `go test` execution finishes. I think we have a bunch of these, but I came across this one during an unrelated troubleshooting. Is it worth fixing this? It's not really causing any issues, it's just sloppy coding. The only way to see any difference is to run the `go test` until it reaches its time limit and panics. In that case, the trace will contain references to the threads. For example: ``` go test github.com/cometbft/cometbft/blocksync -v -run TestBlockPoolMaliciousNode -count 100 -failfast -race -timeout 30s ``` After 30 seconds the test didn't run 100 times yet, hence `go test` panics. Because the test has been run multiple times already, multiple sets of threads will be reported in the panic. With the fix, only one set is reported. Author: @greg-szabo <hr>This is an automatic backport of pull request cometbft#4633 done by [Mergify](https://mergify.com). --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Merge commit from fork lower than what was previously reported GHSA-22qq-3xwm-r5x4 --------- Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Greg Szabo <greg@philosobear.com> Co-authored-by: Greg Szabo <16846635+greg-szabo@users.noreply.github.com> Co-authored-by: Sergio Mena <sergio@informal.systems>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an automatic backport of pull request #2475 done by Mergify.
Cherry-pick of f8366fc has failed:
To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Mergify commands and options
More conditions and actions can be found in the documentation.
You can also trigger Mergify actions by commenting on this pull request:
@Mergifyio refreshwill re-evaluate the rules@Mergifyio rebasewill rebase this PR on its base branch@Mergifyio updatewill merge the base branch into this PR@Mergifyio backport <destination>will backport this PR on<destination>branchAdditionally, on Mergify dashboard you can:
Finally, you can contact us on https://mergify.com