8000 feat: add prefix (directory) support for DAG files by yottahmd · Pull Request #1038 · dagu-org/dagu · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: add prefix (directory) support for DAG files #1038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat: add prefix (directory) support for DAG files #1038

wants to merge 2 commits into from

Conversation

yottahmd
Copy link
Collaborator
@yottahmd yottahmd commented Jun 19, 2025 •
edited
Loading

This PR implements prefix (directory) support for DAG files, allowing users to organize their DAGs in a hierarchical directory structure. Previously, Dagu only supported a flat structure for DAGs. With this change, users can create subdirectories within their DAGs directory and organize workflows logically (e.g., workflow/task1, etl/extract, monitoring/health_check).

The implementation uses the term "prefix" because the DAG store implementation could be object storage or database in the future, not just filesystem.

Changes Made

Phase 1: Core Storage Support

  • Added path utilities (internal/resource/pathutil.go):

    • Path normalization, splitting, joining functions
    • Path validation to ensure security
    • Support for nested directory structures

  • Updated DAG Store interface (internal/models/dag.go):

    • Added ListWithPrefix method to support directory filtering
    • Maintains backward compatibility with existing List method

  • Enhanced file-based storage (internal/persistence/filedag/store.go):

    • Implemented recursive directory traversal
    • Updated CRUD operations to handle prefixed names
    • Automatic parent directory creation
    • Support for renaming across directories

  • Updated DAG model (internal/digraph/dag.go):

    • Added internal fileName field to store full path with prefix
    • FileName() method returns prefixed name when available
    • Maintains backward compatibility for existing DAGs

Phase 2: API Support

  • Updated API schema (api/v2/api.yaml):

    • Modified DAGFileName pattern to support forward slashes
    • Added prefix query parameter to ListDAGs endpoint
    • Added subdirectories response field for directory navigation
    • Updated CreateNewDAG to accept prefixed names

  • Updated API handlers (internal/frontend/api/v2/dags.go):

    • ListDAGs uses ListWithPrefix when prefix parameter provided
    • All endpoints work seamlessly with prefixed DAG names
    • URL encoding/decoding handled automatically by router

Testing

  • Comprehensive unit tests for path utilities and DAG store operations
  • Integration tests demonstrating:
    • Creating/listing DAGs with various prefix levels
    • Directory navigation and filtering
    • Executing prefixed DAGs
    • Backward compatibility

Current Status

Completed:

  • Core storage layer with full prefix support
  • API endpoints updated and tested
  • Backward compatibility maintained

Issues:

  • Scheduler doesn't support subdirectories - Only scans root directory for scheduled DAGs
  • File watcher doesn't monitor subdirectories
  • Scheduled DAGs in subdirectories won't be picked up

Next Steps

  1. Fix scheduler to support subdirectories

    • Update entryReaderImpl to recursively scan directories
    • Fix file watcher to monitor subdirectories
    • Use full DAG names with prefixes

  2. Implement UI changes for directory navigation

    • Add folder icons and navigation
    • Support expanding/collapsing directories
    • Update DAG creation dialog

  3. Update documentation with examples of using prefixes

Examples

# Create DAGs with prefixes
dagu start workflow/daily_report.yaml
dagu start etl/extract/users.yaml
dagu start monitoring/health_check.yaml

# API usage
GET /api/v2/dags?prefix=workflow
GET /api/v2/dags?prefix=etl/extract
POST /api/v2/dags { "name": "workflow/new_task" }

@codecov Codecov
Copy link
codecov bot commented Jun 19, 2025

Codecov Report

Attention: Patch coverage is 81.73077% with 38 lines in your changes missing coverage. Please review.

Project coverage is 68.20%. Comparing base (4ee3c50) to head (4be3512).

Files with missing lines Patch % Lines
internal/persistence/filedag/store.go 70.54% 27 Missing and 11 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1038      +/-   ##
==========================================
+ Coverage   67.77%   68.20%   +0.43%     
==========================================
  Files          93       94       +1     
  Lines       13946    14104     +158     
==========================================
+ Hits         9452     9620     +168     
+ Misses       3687     3675      -12     
- Partials      807      809       +2
Files with missing lines Coverage Δ
internal/digraph/dag.go 72.83% <100.00%> (+4.04%) ⬆️
internal/fileutil/pathutil.go 100.00% <100.00%> (ø)
internal/persistence/filedag/store.go 73.93% <70.54%> (+8.62%) ⬆️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ee3c50...4be3512. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@yottahmd
0