8000 Database structure plans · Issue #2559 · Zilliqa/zq2 · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Database structure plans #2559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
JamesHinshelwood opened this issue Mar 21, 2025 · 0 comments
Open

Database structure plans #2559

JamesHinshelwood opened this issue Mar 21, 2025 · 0 comments
Assignees
@JamesHinshelwood
Labels
Agate Required for mainnet launch

Comments

@JamesHinshelwood
Copy link
Collaborator
JamesHinshelwood commented Mar 21, 2025 •
edited
Loading

A brief overview of our current database structure, our issues with it and my plans to change it. Only the block tables are mentioned below - The other tables are not relevant.

Current Structure (SQLite)

Table Key Value Notes
blocks block_hash (block_without_txns, is_canonical) is_canonical means the block was on the most recently executed fork. It is not related to finalization.
transactions txn_hash txn
receipts txn_hash (receipt, block_hash)

Issues

  1. 'canonical' is not a meaningful property of blocks. It should not exist. It causes confusion and bugs.
  2. It is impossible to represent some valid states of the chain. Multiple unfinalized blocks may exist which contain overlapping transactions, but there is no way to represent this. This is no longer true as of Don't delete transaction receipts when a block is 'reverted' #2588.
  3. Table keys are not efficient. Perhaps this isn't such an issue in SQLite, but generally databases are happier when keys are contiguous sequences of numbers. This is especially true in B-tree-based databases such as redb.

Version 2 Structure (Migration to redb) #2646

Table Key Value Notes
blocks view (block, is_canonical) is_canonical means the block was on the most recently executed fork. It is not related to finalization.
transactions txn_hash txn
receipts txn_hash receipt

This has most of the same issues with the existing structure, but moving the block -> transaction mapping lets us fix issue (2) above in the next version...

Version 2.5 Structure (No migration needed)

Table Key Value Notes
blocks view block Only finalized blocks are stored. Unfinalized blocks are stored in memory.
transactions txn_hash txn Only transactions in finalized blocks are stored.
receipts txn_hash receipt Only receipts in finalized blocks are stored.

This version removes all mentions of 'canonical' from our code base and fixes the transaction-representation issue (i.e. fixes issues (1) and (2)). We can move to this version without any data migration, by just dropping the is_canonical blocks table.

Version 3 Structure (Migration needed)

Table Key Value Notes
headers height header Only finalized blocks are stored. Unfinalized blocks are stored in memory.
txn_indices height (first_txn, txn_count) Stores an index and count into the transactions/receipts table, representing the transactions contained in this block.
aggregate_qcs height [qc] Stores the aggregate QC for a block, as a list of QCs, if there is one.
transactions txn_index txn Only transactions in finalized blocks are stored.
receipts txn_index receipt Only receipts in finalized blocks are stored.

This makes all table keys contiguous integers (fixes issue (3)) and splits blocks into a fixed-length header, plus a variable number of transactions and QCs. It also unlocks large speed ups when syncing, by allowing block headers to be downloaded separately straight into the database and means blocks which don't contain transactions or QCs can be skipped.
@JamesHinshelwood JamesHinshelwood self-assigned this Mar 21, 2025
@DrZoltanFazekas DrZoltanFazekas added the Agate Required for mainnet launch label Apr 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JamesHinshelwood JamesHinshelwood

Labels
Agate Required for mainnet launch
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JamesHinshelwood @DrZoltanFazekas
0