8000 [#938] Add Queue Feature by yottahmd · Pull Request #940 · dagu-org/dagu · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[#938] Add Queue Feature #940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 76 commits into from
May 24, 2025
Merged

[#938] Add Queue Feature #940

merged 76 commits into from
May 24, 2025

Conversation

yottahmd
Copy link
Collaborator
@yottahmd yottahmd commented May 16, 2025

Resolves #938

This PR implements a queue feature that allows user to run workflows concurrently with a limited number of concurrent processes defined in the DAG definition.

To-Do:

  • Local storage for queue items
    • Functionality for directory management for each named queues
    • Priority Queue (Priority & Standard)
    • Queue Item (contains workflow ID)
  • Workflow status for Queued
  • Queue Manager to provide an interface to interact with queues
  • Queue worker process inside the scheduler service
  • Communication path via unix socket to queue processor to make queueing fast
  • Commands for queue and dequeue
  • API to list queue items GET /queues{name}
  • API to enqueue an item POST /queue/{name}
  • API to dequeue an item DELETE /queue/{name}/{id}
  • WebUI to list queue information
  • WebUI to enqueue / dequeue a workflow execution

@yottahmd yottahmd changed the base branch from main to next May 16, 2025 05:20
@yottahmd yottahmd self-assigned this May 16, 2025
@yottahmd yottahmd force-pushed the 938-queue-feature branch from 394f1c3 to 42135ca Compare May 23, 2025 16:31
@yottahmd yottahmd marked this pull request as ready for review May 24, 2025 06:36
@yottahmd yottahmd merged commit 2fa7250 into next May 24, 2025
4 checks passed
@yottahmd yottahmd deleted the 938-queue-feature branch May 24, 2025 06:40
Copy link
codecov bot commented May 24, 2025

Codecov Report

Attention: Patch coverage is 62.67373% with 564 lines in your changes missing coverage. Please review.

Project coverage is 59.98%. Comparing base (e8b7ab4) to head (1c35fce).
Report is 1 commits behind head on next.

Files with missing lines Patch % Lines
...nternal/persistence/localqueue/prototype/reader.go 62.40% 84 Missing and 19 partials ⚠️
internal/scheduler/scheduler.go 40.25% 91 Missing and 4 partials ⚠️
...rnal/persistence/localqueue/prototype/queuefile.go 66.15% 43 Missing and 23 partials ⚠️
internal/persistence/localdag/store.go 20.00% 47 Missing and 1 partial ⚠️
internal/persistence/localproc/handle.go 58.42% 30 Missing and 7 partials ⚠️
internal/persistence/localqueue/prototype/store.go 69.15% 22 Missing and 11 partials ⚠️
internal/cmd/dequeue.go 56.25% 19 Missing and 9 partials ⚠️
internal/cmd/enqueue.go 63.01% 18 Missing and 9 partials ⚠️
internal/persistence/localqueue/prototype/queue.go 70.32% 19 Missing and 8 partials ⚠️
internal/persistence/localproc/procgrp.go 62.12% 17 Missing and 8 partials ⚠️
... and 14 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             next     #940      +/-   ##
==========================================
+ Coverage   59.85%   59.98%   +0.12%     
==========================================
  Files          70       83      +13     
  Lines        8953    10168    +1215     
==========================================
+ Hits         5359     6099     +740     
- Misses       3096     3467     +371     
- Partials      498      602     +104     
Files with missing lines Coverage Δ
internal/cmd/dry.go 78.04% <100.00%> (+0.27%) ⬆️
internal/cmd/flags.go 100.00% <ø> (ø)
internal/cmd/retry.go 73.40% <100.00%> (+2.57%) ⬆️
internal/cmd/stop.go 72.72% <100.00%> (ø)
internal/config/config.go 76.74% <ø> (ø)
internal/config/loader.go 90.05% <100.00%> (+0.58%) ⬆️
internal/digraph/dag.go 72.41% <ø> (ø)
internal/fileutil/fileutil.go 33.69% <100.00%> (ø)
internal/models/history.go 0.00% <ø> (ø)
internal/persistence/localqueue/prototype/job.go 100.00% <100.00%> (ø)
... and 27 more

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8b7ab4...1c35fce. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

yottahmd added a commit that referenced this pull request May 24, 2025
Original implementation by Kriyanshi: #690

Co-authored-by: Kriyanshi Shah <59494745+kriyanshii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement Queue Feature
1 participant
0