feat: car export supports custom dag traversal and export #767
Conversation
packages/car/src/index.ts
Outdated
| const isTargetRoot = roots.some(r => r.equals(lastCid)) | ||
| if (!isTargetRoot) { | ||
| throw new Error('knownDagPath must end with one of the target roots') | ||
| } |
There was a problem hiding this comment.
we need some solution here to handle multiple roots where there are paths to each root.
I know this sounds confusing, but "root" according to current kubo/boxo is the "target" root block of a dag.. the "dagRoot" is the root of the dag walked to get to that subdag starting node
|
cspell is yelling about some words in dag-scope.spec.ts |
|
ping @achingbrain @2color I think some other eyes on this would be good, and this is blocking ipfs/service-worker-gateway#356 |
Co-authored-by: Alex Potsides <alex@achingbrain.net>
|
Ok @achingbrain I've got things refactored to use the traversal and export strategy API you suggested. One thing I don't like about these changes is that we have to do traversal twice, but if root===target or target===some-leaf-node, it's practically no extra work on either side. Let me know if you see anything that can be improved. |
| export class BlockExporter implements ExportStrategy { | ||
| async * export (cid: CID, block: BlockView<any, any, any, 0 | 1>): AsyncGenerator<CID, void, undefined> { | ||
| // don't yield the block, index.ts will add it to the car file and then | ||
| // we're done |
Title
feat: support dag-scope
Description
This PR adds support for the
dag-scopequery parameter to the export and stream methods. This parameter allows the user to specify the scope of the DAG they want to retrieve. The possible values areall,entity, andblock. The default value isall.Also refactored the code quite a bit to support different strategies for walking the dag, and added two new options: dagRoot & knownDagPath.
This code also updates the
blockFiltercheck to occur before theblockstore.getcall to prevent potentially unnecessary network calls.Related ipfs/helia-verified-fetch#198
Related ipfs/service-worker-gateway#356
Notes & open questions
Change checklist