8000 [Computer] Add App-Use to Computer interface by ddupont808 · Pull Request #280 · trycua/cua · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Computer] Add App-Use to Computer interface #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 41 commits into from
May 31, 2025
Merged

[Computer] Add App-Use to Computer interface #280

merged 41 commits into from
May 31, 2025

Conversation

ddupont808
Copy link
Contributor
AppUse.mp4

Introduces App-Use, an experimental feature that creates lightweight virtual desktops scoped to specific applications, enabling focused agent control and parallel multi-agent workflows.

Changes

  • Added experiments parameter to Computer.__init__()
  • Added create_desktop_from_apps() method to create app-scoped desktops
  • Added assertion requiring experiments=["app-use"] to enable feature
  • Created blog post documenting usage and examples

Example

# Enable App-Use and create app-scoped desktop
computer = Computer(experiments=["app-use"])
desktop = computer.create_desktop_from_apps(["Safari", "Notes"])

# The app-scoped desktop exposes a Computer interface
# This screenshot will only show Safari and Notes, and will be dynamically scaled to fit the app windows
screenshot = await desktop.interface.screenshot()
with open("screenshot.png", "wb") as f:
    f.write(screenshot)

# Agent can now only see/interact with Safari and Notes
agent = Agent(computer=desktop, model=LLM(...))
await agent.run("Research AI news and take notes")

Dillon DuPont added 30 commits May 11, 2025 17 8000 :35
add get_keyboard_focus to computer_server
fffe51b
include selected text range
10ba905
fix
692cd6c
added AXFocused attribute
bc8f0d9
added error logging
52bf135
error logging
28eaf43
Fixes issue #172
9adfb93
Fixes issue #172
4509121
Merge branch 'main' into feature/computer/extensions
a53c1a4
remove kb focus func
45b011d
8000
working macOS desktop multiplexer
92101ab
added dock and menubar bounds
a6a565e
2 pass dock rendering
c1ab6b8
removed class
ab3f286
working smart resize
1d273da
set minimum size
85c365d
demo mode
a986062
light refactor
67c187d
refactoring
da6917f
fixed full capture
653ff1c
virtual desktop abstraction
b5a87f8
hitbox mapping output
90ec3bd
refactor focus context
a539000
fixed hitbox logic
c8b3ce1
agent example
5ccaa5f
Merge branch 'main' into feature/computer/extensions
5063477
fixed hitboxes and agent example
2c04721
Merge branch 'main' into feature/computer/extensions
0519c50
removed selected text attr
10b430e
added diorama to server side
c510eb3
@ddupont808 ddupont808 merged commit 99b979e into main May 31, 2025
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@f-trycua f-trycua f-trycua approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ddupont808 @f-trycua
0