10000 [epic]: graph db: SQL implementation & migration · Issue #9795 · lightningnetwork/lnd · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[epic]: graph db: SQL implementation & migration #9795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 of 60 tasks
ellemouton opened this issue May 8, 2025 · 0 comments
Open
2 of 60 tasks

[epic]: graph db: SQL implementation & migration #9795

ellemouton opened this issue May 8, 2025 · 0 comments
Assignees
@ellemouton
Labels
epic Issues created to track large feature development graph migration sql
Milestone
v0.20.0

Comments

@ellemouton
Copy link
Collaborator
ellemouton commented May 8, 2025 •
edited
Loading

This is the tracking issue for the graph db SQL-ization project. It will be the go-to reference to see how the project is progressing.

I'll start with a check-list for easy reference & then will add any details & context lower down behind some drop-down tags.

High Level Project Check List

  • Prepare the code (exported methods and tests) such that everything is DB agnostic (ie, hide db specific types and behaviour) in preparation for having things ready to be run against a different DB backend.
  • Introduce an abstract graph DB interface & let the ChannelGraph use this interface instead of a raw pointer.
  • Introduce SQL schema, queries & CRUD
  • Implement the kvdb -> SQL migration for the above stores.
  • Plug the new migration into the main code base & add itest jobs to run against them (behind a feature flag s.t they are not yet available in prod build).
  • Once there is enough confidence in the code & migration, add the SQL impl & migrations to the main prod build

Lower Level Check Lists

1. Prep graph DB code to be ready for new db backend

2. Abstract graph DB interface

3. Introduce SQL schema, queries & CRUD

We'll define the schemas, queries and implement the crud required for each of the following data strucutres:

  • nodes
  • channels
  • channel updates
  • zombies
  • source nodes
  • closed SCIDs
  • prune log

Note that this will be done in an incremental fashion s.t we are implementing the SQL impl of the new abstract graph store interface method by method. Here are all the methods that will be converted:

  • AddLightningNode
  • AddrsForNode
  • ForEachSourceNodeChannel
  • ForEachNodeChannel
  • ForEachNodeCached
  • ForEachNode
  • ForEachNodeCacheable
  • LookupAlias
  • DeleteLightningNode
  • NodeUpdatesInHorizon
  • FetchLightningNode
  • HasLightningNode
  • IsPublicNode
  • GraphSession
  • ForEachChannel
  • DisabledChannelIDs
  • AddChannelEdge
  • HasChannelEdge
  • DeleteChannelEdges
  • AddEdgeProof
  • ChannelID
  • HighestChanID
  • ChanUpdatesInHorizon
  • FilterKnownChanIDs
  • FilterChannelRange
  • FetchChanInfos
  • FetchChannelEdgesByOutpoint
  • FetchChannelEdgesByID
  • ChannelView
  • MarkEdgeZombie
  • MarkEdgeLive
  • IsZombieEdge
  • NumZombies
  • PutClosedScid
  • IsClosedScid
  • UpdateEdgePolicy
  • SourceNode
  • SetSourceNode
  • PruneTip
  • PruneGraphNodes
  • PruneGraph
  • DisconnectBlockAtHeight

Once all the above is complete we can:

  • run the full uint test suite against the SQL graph impls
  • run the full itest suite against the SQL graph impls

4. Migration

- write the migration code for the graph store along with a test for it - plug in to code behind a dev feature flag

Planned Schemas:

Nodes

NOTES:

  1. the node_v2_data is only for demonstrating how future gossip V2 data will fit in to the picture
  2. Still up for debate if we want to merge the node_v1_data and node_v2_data tables into the nodes table.

Image

Channels

NOTES:

  1. the channels_v2_data is only for demonstrating how future gossip V2 data will fit in to the picture
  2. Still up for debate if we want to merge the channels_v1_data and channels_v2_data tables into the channels table.
  3. The nodes table here is not complete - it is just to show how channels link to nodes. See "Nodes" section above for details on the nodes table.

Image

Channel Policies

  1. the channel_policy_v2_data is only for demonstrating how future gossip V2 data will fit in to the picture
  2. Still up for debate if we want to merge the channel_policy_v1_data and channel_policy_v2_data tables into the channel_policies table.
  3. The nodes and channels tables here are not complete - it is just to show how channel_policies link to channels and to nodes. See "Nodes" & "Channels" sections above for details on the nodes and `channels tables.

Image

Other Graph Tables

Image

@ellemouton ellemouton added this to the v0.20.0 milestone May 8, 2025
@ellemouton ellemouton self-assigned this May 8, 2025
This was referenced May 8, 2025
Merged
Merged
Merged
Merged
Merged
Merged
@saubyk saubyk added this to lnd v0.20 May 8, 2025
@saubyk saubyk moved this to In progress in lnd v0.20 May 8, 2025
@saubyk saubyk added the epic Issues created to track large feature development label May 8, 2025
@ellemouton ellemouton mentioned this issue May 9, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ellemouton ellemouton

Labels
epic Issues created to track large feature development graph migration sql
Projects
lnd v0.20
Status: In progress
Milestone
v0.20.0
Development

No branches or pull requests
2 participants
@ellemouton @saubyk
0