8000 KIP-111 Prune obsolete trie nodes by blukat29 · Pull Request #1871 · klaytn/klaytn · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on Aug 19, 2024. It is now read-only.

KIP-111 Prune obsolete trie nodes #1871

Merged
merged 12 commits into from
Jul 26, 2023
Merged

KIP-111 Prune obsolete trie nodes #1871

merged 12 commits into from
Jul 26, 2023

Conversation

blukat29
Copy link
Contributor
@blukat29 blukat29 commented Jun 26, 2023 •
edited
Loading

Proposed changes

See KIP-111 for the background.

Usage:

# init custom genesis and sync
kcn init --state.live-pruning genesis.json
kcn   # after init, --state.live-pruning is optional

# cypress sync
ken --state.live-pruning  # first run, when database is clean.
ken   # from the second run, --state.live-pruning is optional

# temporarily disable live pruning
ken --state.live-pruning-retention 0

Changes:

  • New database schema

    • (MiscDB) PruningEnabled: ["PruningEnabled"] -> true is a database-stored flag that tells the trie nodes are in prunable format (i.e. with ExtHash).
    • (MiscDB) PruningMark: ["Pruning-" + blockNum + ExtHash] -> true marks an obsolete trie node that's not yet deleted.

  • New CLI flags

    • kcn init --state.live-pruning
      • if specified, write PruningEnabled to database.
    • kcn --state.live-pruning
      • if specified and database is clean, write PruningEnabled to database.
      • After the first writing PruningEnabled, this flag becomes optional.
    • kcn --state.live-pruning-retention <172800>
      • Number of blocks to keep obsolete nodes before deleting. Defaults to 2 days (172800).
      • Set 0 to disable live pruning.
      • Ignored if the database has pruning disabled.

  • New struct fields

     cn.Config.LivePruning
 cn.Config.LivePruningRetention
 blockchain.CacheConfig.LivePruningRetention
 statedb.TrieOpts.PruningBlockNumber uint64        // replaces  TrieOpts.Pruning bool

  • Pruning actions

    • Trie.insert and Trie.delete marks obsolete nodes as PruningMarks, only if:
      • PruningEnabled in database AND
      • The Trie is created with PruningBlockNumber != 0 (i.e. through PrunableStateAt())
    • blockchain.insertChain uses PrunableStateAt()
    • worker.makeCurrent uses PrunableStateAt()
    • blockchain.writeStateTrie asynchronously deletes trie nodes according to PruningMarks.

Types of changes

  • Bugfix
  • New feature or enhancement
  • Others

Checklist

  • I have read the CONTRIBUTING GUIDELINES doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes ($ make test)
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

Related issues

Further comments

@blukat29 blukat29 self-assigned this Jun 26, 2023
@blukat29 blukat29 changed the title Prune obsolete trie nodes KIP-111 Prune obsolete trie nodes Jun 26, 2023
@blukat29 blukat29 force-pushed the trie-pruning-queue branch 3 times, most recently from e6da165 to 44b6908 Compare June 30, 2023 03:45
@blukat29 blukat29 mentioned this pull request Jun 30, 2023
Merged
9 tasks
@blukat29 blukat29 force-pushed the trie-pruning-queue branch 3 times, most recently from 75e6ef1 to 68dc336 Compare July 4, 2023 09:27
@blukat29 blukat29 force-pushed the trie-pruning-queue branch from fd4e25c to 1709442 Compare July 17, 2023 04:45
@blukat29 blukat29 marked this pull request as ready for review July 17, 2023 04:55
@blukat29
Copy link
Contributor Author
blukat29 commented Jul 19, 2023 •
edited
Loading

Cypress chain import performance test

  • Used admin_importChain with Cypress blocks 0 to 50,000,000
  • chaindata size overall 70% down = statetrie 87% down, miscdb 200% up
  • insertion time overall 30% down = processTxs 50% down, others 20~30% up

@aidan-kwon aidan-kwon added the need to merge Need to merge for the next time label Jul 24, 2023
@JayChoi1736 JayChoi1736 mentioned this pull request Jul 24, 2023
Closed
20 tasks
aidan-kwon
aidan-kwon reviewed Jul 24, 2023
View reviewed changes
@blukat29
Copy link
Contributor Author

@aidan-kwon Fixed the potential concurrency issue regarding insertPruningMark(). The treatment is something similar to what insertPreimage() does. I think the performance impact is not too serious because db.lock is acquired only at Trie.Commit(). Trie.Commit() happens [1 + number of contract accounts changed] times per block.

@aidan-kwon
Copy link
Member

@2dvorak @hyunsooda @yoomee1313 PTAL

2dvorak
2dvorak approved these changes Jul 26, 2023
View reviewed changes
@blukat29 blukat29 mentioned this pull request Jul 26, 2023
Merged
9 tasks
@blukat29 blukat29 merged commit 5eb2c53 into klaytn:dev Jul 26, 2023
@blukat29 blukat29 deleted the trie-pruning-queue branch July 26, 2023 05:27
@blukat29 blukat29 mentioned this pull request Jul 26, 2023
Merged
9 tasks
hyunsooda
hyunsooda reviewed Jul 27, 2023
View reviewed changes
Copy link
Contributor
@hyunsooda hyunsooda left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Learned a good implementation. For the minor question, How did you draw the chart? Did you just simply lookup the directory size?

@aidan-kwon aidan-kwon mentioned this pull request Jul 30, 2023
Closed
9 tasks
@blukat29 blukat29 mentioned this pull request Jul 31, 2023
Merged
9 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@hyunsooda hyunsooda hyunsooda left review comments

@jeongkyun-oh jeongkyun-oh jeongkyun-oh approved these changes

@ian0371 ian0371 ian0371 approved these changes

@2dvorak 2dvorak 2dvorak approved these changes

@aidan-kwon aidan-kwon aidan-kwon approved these changes

@yoomee1313 yoomee1313 yoomee1313 approved these changes

@ethan-kr ethan-kr Awaiting requested review from ethan-kr

@kjeom kjeom Awaiting requested review from kjeom kjeom is a code owner

Assignees

@blukat29 blukat29

Labels
need to merge Need to merge for the next time
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@blukat29 @aidan-kwon @ian0371 @hyunsooda @2dvorak @jeongkyun-oh @yoomee1313
0