What Computer Use Actually Is
Computer use means the agent can see screenshots and generate mouse and keyboard inputs. It is screen sharing plus remote control.
The technical mechanism varies by implementation, but the core is the same. The agent receives screenshots of your display, either continuously or on demand. It analyzes those screenshots to understand what is on screen — windows, buttons, text fields, application states. Based on that understanding, it generates mouse movements, clicks, and keyboard inputs that are executed on your system. The effect is that the agent can do anything you can do with a mouse and keyboard.
In Claude Code, computer use ships as a built-in MCP server that is off by default. You enable it through the /mcp command, and then you grant your terminal Accessibility and Screen Recording permissions in macOS System Settings. These are the same permissions you would grant to any screen-sharing or automation tool. Without them, the agent cannot see or control your screen.
In the ChatGPT desktop app, computer use is a plugin that works in Work and Codex modes. It runs on macOS and Windows in supported regions. The architecture is different — it runs within the desktop app rather than from a CLI — but the capability is the same: see the screen, generate inputs.
What This Solves
Developer work is not entirely terminal-based. You use browsers to test web applications, design tools to inspect mockups, database GUIs to explore data, cloud consoles to configure infrastructure. These tools often have CLI equivalents, but sometimes they do not, and sometimes the GUI is faster even when a CLI exists. Without computer use, you had to switch to those tools yourself and describe what you saw back to the agent. The context break was friction.
With computer use, the agent can interact with the GUI directly. It can open a browser, navigate to a URL, inspect what rendered, and compare it to what was expected. It can fill out a form to test validation. It can click through a setup wizard. It can take a screenshot of a bug and annotate what is wrong. These are small tasks, but they add up, and eliminating the context switch between agent work and manual GUI work is a genuine quality-of-life improvement.
The capability also extends beyond developer tools. An agent with computer use can interact with any application on your system. It can send a message in Slack, create a document in Google Docs, update a row in a spreadsheet. If the task involves a GUI application and you would rather not do it yourself, the agent can now do it.
What This Creates
The same capability that lets the agent help with GUI tasks lets it see everything on your screen.
When you grant screen recording permission, you are not granting it for one application. You are granting it for everything visible on your display. The agent that is supposed to be testing your web application can also see your email in the background, your Slack messages in the corner, your calendar notification that just popped up. It can see passwords if they are visible. It can see private messages if they are on screen. It can see anything that is not hidden.
The permissions are binary. You grant Accessibility permission and the agent can control inputs. You grant Screen Recording permission and the agent can see everything. There is no granular control that says only look at this window or only control this application. The system-level permissions are all-or-nothing, and the trust you are placing is correspondingly broad.
This is not a flaw in the design. It is a consequence of how screen sharing works at the operating system level. But it means that enabling computer use is a significant decision. You are not just letting the agent interact with the tool you are working on. You are letting it see your whole display, all the time it is active, including things you might prefer it did not see.
| Permission | What it grants | Scope |
|---|---|---|
| Screen Recording | See all visible windows | Everything on all displays |
| Accessibility | Generate mouse and keyboard input | System-wide |
| Neither | Cannot use computer use | Feature disabled |
The Operational Reality
Computer use in current implementations is a research preview, not a production feature. The documentation is explicit about this. It works well for some tasks and poorly for others. It can be slow — taking screenshots, analyzing them, deciding what to do — compared to the instant execution of a CLI command. It can misclick, misread text, or get confused by unfamiliar interfaces. It is impressive when it works and frustrating when it does not.
The recommendation from vendors is to use computer use for specific tasks where it is needed, not to leave it enabled by default. Run your terminal work as you always have. When you hit a task that requires GUI interaction, enable computer use, let the agent do the task, verify the result, and disable it again. This limits the window during which the agent can see your screen, and it forces a deliberate choice each time the capability is used.
The practical setup also matters. If you are going to enable computer use, close applications you do not want the agent to see. Hide sensitive windows. Keep your display clean. The agent cannot see what is not visible, so controlling what is visible is a way to control access. This is manual and imperfect, but it is the operational reality until more granular permissions exist.
What to Consider Before Enabling
Computer use is useful. The permission grant is broad. Think before you flip the switch.
- Understand the scope. Screen Recording permission lets the agent see everything on your display. Accessibility permission lets it control inputs system-wide. There is no window-level granularity.
- Clear your screen before enabling. Close or hide applications with sensitive content — email, messages, password managers, financial tools. The agent cannot see what is not visible.
- Use it for specific tasks, then disable. Do not leave computer use enabled while you work on unrelated things. Enable when needed, disable when done.
- Verify what it did. After the agent completes a GUI task, check the result yourself. Computer use can misclick or misunderstand. Trust but verify.
- Consider your work environment. If your screen routinely shows confidential information — client data, internal communications, credentials — computer use may not be appropriate for your workflow.
- Read the documentation. Both Claude Code and Codex have specific guidance about computer use, its limitations, and its appropriate use cases. The vendors are clearer about the risks than most.
Mindly keeps your knowledge in local files, so your second brain does not need screen access to be useful. A private approach →
The Direction This Is Going
Computer use is in research preview today. It will be a standard feature soon. The trajectory is clear: AI agents are moving from text environments to graphical environments, from running commands to operating interfaces, from being tools you invoke to being assistants that can do what you would do if you had the time. The terminal was a natural boundary. It is becoming a choice.
The permissions model may evolve. Operating systems could add more granular screen sharing controls, letting you grant access to specific applications or windows rather than the entire display. Vendors could build isolation features, running computer use in a sandboxed environment that cannot see your main desktop. These would reduce the access tradeoff. They do not exist today.
In the meantime, computer use is a capability that exists, works imperfectly, and requires a trust decision. If the tasks you need justify the access, the feature is genuinely useful. If you are uncomfortable with an agent seeing your whole screen, the feature can stay off. The choice is yours, which is the right way for a capability this broad to be introduced.