8000 Community call 1/11/2024: Databases & db perf · Issue #2023 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Community call 1/11/2024: Databases & db perf #2023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
faddat opened this issue Jan 11, 2024 · 4 comments
Closed

Community call 1/11/2024: Databases & db perf #2023

faddat opened this issue Jan 11, 2024 · 4 comments
Assignees
@adizere
Labels
community-call To be discussed in an upcoming community call P:storage-optimization Priority: Give operators greater control over storage and storage optimization
Milestone
2024-Q1

Comments

@faddat
Copy link
Contributor
faddat commented Jan 11, 2024 •
edited
Loading

Please correct any of these items if they're wrong:

  • sqlite is off the table for now
  • there's a desire to move to a single kv store:
    • goleveldb - informal will optimize and champion this kv store
      • Pros
        • most chains already use this
      • Cons
        • goleveldb seems to have a scary performance drop as the number of keys and values increases
    • pebbledb - notional will optimize and champion this kv store

      • Pros

        • seems fastest in the widest variety of real world situations

      • Cons

        • when there are not many keys or the values or small, goleveldb is faster

      • for users of rocks, the migration is built into pebble (I've done it and it works but of course this is not a guarantee)

  • badger, bolt, and cleveldb are safe to drop but only if there's a migration path because we don't want to disturb legacy users
  • flaky test detection work was helpful and if I did stuff like that in the future, that'd be a good thing
  • query best way to test relative db perf, in an exclusively comet context? (eg: instead of doing benchmarks only in the comebft-db library)

thanks and again -- if any of that is wrong -- please let me know
@adizere adizere added community-call To be discussed in an upcoming community call P:storage-optimization Priority: Give operators greater control over storage and storage optimization labels Jan 12, 2024
@adizere
Copy link
Member
adizere commented Jan 12, 2024 8000

This is excellent! I'd only add that we care about having easily reproducible and comparable results. So we want to tackle #63 towards being able to make a clear assessment of "X is best than Y as a storage backend". More specifically, first step would likely be this one: #46

@faddat
Copy link
Contributor Author
faddat commented Jan 14, 2024

thanks for the feedback!

@adizere adizere self-assigned this Jan 19, 2024
@adizere adizere added this to CometBFT Jan 19, 2024
@adizere adizere added this to the 2024-Q1 milestone Jan 19, 2024
@github-project-automation github-project-automation bot moved this to Todo in CometBFT Jan 19, 2024
@adizere
Copy link
Member
adizere commented Mar 21, 2024

Hey @faddat did you have a look at the recent report on storage we wrote ? We introduced this in #2596.

The report is not so much about DB performance, but storage efficiency.

@adizere
Copy link
Member
83F3 adizere commented Apr 1, 2024

Closing with #2596. We need to consider what would be good next steps.

@adizere adizere closed this as completed Apr 1, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in CometBFT Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adizere adizere

Labels
community-call To be discussed in an upcoming community call P:storage-optimization Priority: Give operators greater control over storage and storage optimization
Projects
No open projects
CometBFT
Status: Done
Milestone
2024-Q1
Development

No branches or pull requests
2 participants
@adizere @faddat
0