Claude Code Computer Use Pitfalls | CLI Integration Limitations

What Is Claude Code Computer Use — The Last Resort Among 4 Tools

Claude Code Computer Use is a mechanism that delegates mouse, keyboard, and screen recognition to Claude from within a Claude Code CLI session. It is implemented as a built-in MCP server named computer-use, disabled by default, and only activates in projects where it has been enabled via the /mcp menu (Source: Let Claude use your computer from the CLI).

Crucially, in its official documentation, Anthropic positions Computer Use as the "broadest and slowest" tool — designed to be the last option Claude considers among available choices. The priority order is explicitly stated as follows:

In other words, Computer Use is a last resort reserved for situations that are unreachable by other means — such as native macOS apps, iOS Simulators, design tools, or tools without APIs. Conversely, using Computer Use for tasks that can be completed via CLI or via services with existing MCP servers is a design mistake that incurs slowness and a broad security boundary with no benefit in return.

Differences from the Desktop App Version

The Claude Code CLI and the Claude Desktop app share the same Computer Use engine, but differ in UI and some features (Source: Let Claude use your computer from the CLI).

Desktop sessions launched from Dispatch (Claude's cloud execution platform) can invoke Computer Use, but the concept of Dispatch-based execution simply does not apply to Claude Code CLI sessions. If you need to run Computer Use on the cloud side, choose Desktop + Dispatch.

4 Requirements — Mac / Pro·Max / v2.1.85 / Interactive

Unlike other Claude Code features, Computer Use has strictly limited requirements. All 4 must be satisfied for computer-use to appear in the /mcp menu (Source: Let Claude use your computer from the CLI).

There is also an implicit requirement: authentication must go through claude.ai. If you are accessing Claude via a third party such as Amazon Bedrock, Google Cloud Vertex AI, or Microsoft Foundry, you will need a separate claude.ai account just for Computer Use.

Why Team and Enterprise Plans Cannot Use It

It may seem counterintuitive that Pro and Max plans support Computer Use while Team and Enterprise do not. This is because Computer Use is still in the research preview stage, with verification focused on individual user use cases. Having Claude directly operate a corporate machine under an organization account raises many issues around enterprise security controls and consistency, so the feature rollout is proceeding carefully.

For those who need Computer Use on Team or Enterprise plans, the practical options at this time are to switch to a personal Pro or Max account or use the Desktop app. The Computer Use roadmap does not rule out expansion to Team and Enterprise, so keep checking your plan status with /status while you wait.

Why It Cannot Be Used in -p Non-Interactive Mode

Claude Code offers a one-shot non-interactive mode via claude -p "prompt", but Computer Use is not available through this path. The reason is that Computer Use requires interactive safety mechanisms during turns, such as macOS notifications, dialogs, and Esc key handling — and it is considered too risky when no user is watching, as in CI/CD or batch execution scenarios.

For requirements that need GUI automation in CI/CD or automated environments like GitHub Actions, it is more practical to call existing automation tools such as Playwright, AppleScript, or Shortcuts via Bash. Understand Computer Use as a feature that fundamentally assumes "an interactive session with a human watching the screen."

5-Minute Setup — /mcp Activation and 2 macOS Permissions

Once the requirements are met, setup is complete in 3 steps. The actual time is under 5 minutes, and including the System Settings launch for granting permissions, it won't exceed 10 minutes (Source: Let Claude use your computer from the CLI).

# Step 1: Open the MCP menu in an interactive session
$ claude
> /mcp
# Find `computer-use` in the server list (shown as disabled)

# Step 2: Select `computer-use` → Enable
# Settings are retained per project, so this only needs to be done once per project

# Step 3: On first use, macOS will prompt for 2 permissions:
#   - Accessibility (click, input, scroll)
#   - Screen Recording (screen recognition)
# Open System Settings from the prompt link and grant both

Right after granting Screen Recording, macOS may require a restart of the requesting process. Fully quit Claude Code, relaunch it, navigate back to the same prompt, and proceed using the Try again button. If the prompt keeps repeating, visually verify in System Settings > Privacy & Security > Screen Recording that the Claude Code entry is enabled.

After setup, you can request GUI-related tasks in natural language.

Build the app target, launch it, and click through each tab to make
sure nothing crashes. Screenshot any error states you find.

Checklist When computer-use Does Not Appear in /mcp

If computer-use is not shown in the /mcp list, check the 4 requirements in order:

1. Is claude --version 2.1.85 or higher? (Older versions don't show MCP at all)
2. Is a Pro or Max plan active? (Check with /status)
3. Are you running on macOS? (Not shown on Linux or WSL)
4. Are you launching claude without flags? (Via -p is not eligible)

If all of the above are satisfied and it still doesn't appear, check whether you are authenticated via claude.ai. If you are only authenticated via a third party like Bedrock, the Computer Use feature flag is not attached to your account, and you will need to link a separate claude.ai account.

Safe Operation — 3-Tier Control and Per-App Session Approval

Computer Use directly accesses approved apps on your real desktop — unlike sandboxed Bash tools. To reduce the risks of prompt injection (an attack where malicious instructions on screen are fed to Claude to hijack the session) and unintended operations, Anthropic has built in a 5-layer set of guardrails (Source: Let Claude use your computer from the CLI).

Per-App Session Approval and Sentinel Warnings

Simply enabling computer-use does not grant access to all apps on your machine. When Claude needs a new app during a session, a prompt will appear in the terminal showing the app name and additional permissions. You choose Allow for this session or Deny, and approval is valid only within the current session.

For apps with broad access scope, a sentinel warning is displayed before approval:

These apps are not prohibited — the warnings are designed to let humans judge whether the task actually requires this level of access. For example, a task that simply says "take a screenshot of this Slack thread" has no reason to approve Finder access, so you can safely Deny it by default.

3 Tiers of Control Levels

Even for approved apps, Claude's level of control is automatically limited based on the app category:

The design decision not to grant Full control for browsers is to prevent accidents where Claude misidentifies a phishing site and clicks a money transfer button on a financial service. If browser automation is your actual use case, the correct answer is to use Claude in Chrome, which is priority 3.

Esc Global Stop and Lock File

The moment Claude takes control of the screen, a macOS notification appears: "Claude is using your computer · press Esc to stop." Pressing Esc from anywhere immediately stops the current action, and Ctrl+C in the terminal has the same effect. Esc key presses are consumed by Claude, which also prevents malicious use of prompt injection to dismiss dialogs.

Additionally, Computer Use holds a machine-wide lock while active. If another Claude Code session is already using the computer, a new attempt will fail with the message "Computer use is in use by another Claude session." Terminate the session holding the lock first. If the session terminates abnormally due to a crash, Claude will detect the absence of the process and automatically release the lock.

Furthermore, while Claude is in control, the terminal window is automatically excluded from screenshots. This blocks the path by which malicious prompts displayed on screen could be fed back to Claude to hijack the session.

4 Use Cases — Native Builds / E2E / Visual / GUI-Driven

The true strength of Computer Use is that it can bring any work that would otherwise require "leaving the terminal and doing it manually" into the CLI session. The official documentation highlights 4 representative workflows (Source: Let Claude use your computer from the CLI).

Native App Build and Verification

After building native macOS or iOS apps (Swift / Objective-C) with xcodebuild, the step of launching the app and verifying the GUI works tends to remain manual. With Computer Use, you can automate this entire process with a single prompt.

Build the MenuBarStats target, launch it, open the preferences window,
and verify the interval slider updates the label. Screenshot the
preferences window when you're done.

Claude will run xcodebuild via Bash, launch the app after a successful compile, open the preferences window, move the slider, and visually verify the label update. It returns a screenshot showing where things failed, or a verified screenshot on success.

End-to-End UI Testing

You can cover an entire onboarding flow for an Electron app or any native app with a single prompt. No Playwright setup or test harness required — a natural language instruction at the level of "screenshot every screen from sign-up to completion" is all you need.

Open the iOS Simulator, launch the app, tap through the onboarding
screens, and tell me if any screen takes more than a second to load.

GUI-only tools like the iOS Simulator are controlled by Claude with the same feel as using a mouse. For teams that have spent test effort maintaining E2E scripts, this automation is well worth considering.

Reproducing Visual Bugs and Layout Issues

You can have Claude reproduce bugs that only a human would notice by looking — like CSS bugs that only appear at a specific window size.

The settings modal clips its footer on narrow windows. Resize the app
window down until you can reproduce it, screenshot the clipped state,
then check the CSS for the modal container.

Claude resizes the window while searching for the conditions that trigger the bug, takes a screenshot when reproduced, reads the relevant CSS, and proposes a hypothesis for the cause. "Pixel-level visual bugs" were difficult to handle with AI pair programming, but Computer Use tackles them head-on.

Driving GUI-Only Tools

Claude can drive GUI-only tools that have no CLI or API — design software, hardware control panels, iOS Simulators, in-house proprietary tools — even when there are absolutely no other automation options available. As long as a human can operate it with a mouse, Computer Use can replicate those operations.

Conversely, when a CLI, API, or MCP connector is available, never choose Computer Use — that is the golden rule. Choosing Computer Use over priority 1–3 tools when they are available means accepting slowness, a broad security boundary, and increased screenshot token consumption for zero benefit.

Summary — Criteria for Choosing Computer Use

Claude Code Computer Use is a research preview feature that lets Claude directly operate the macOS GUI, delivering irreplaceable value in 4 use cases: native app builds, E2E UI testing, visual bug reproduction, and driving GUI-only tools. At the same time, Anthropic itself positions it as "the last resort", and the principle is to try MCP, Bash, or Claude in Chrome first whenever they can be used.

The pre-use checklist has 4 items:

• Running on macOS (Linux and Windows are not supported)
• Subscribed to Pro or Max plan (Free, Team, and Enterprise are not supported)
• Running Claude Code v2.1.85 or later
• Launched in an interactive session (no -p flag)

Setup is complete simply by enabling computer-use in /mcp and granting two macOS permissions — Accessibility and Screen Recording. During operation, a 5-layer set of safety mechanisms is in place: per-app session approval, 3-tier control, Esc global stop, lock file, and screenshot exclusion.

As this is a research preview, API formats and plan requirements may change in the future. Check Let Claude use your computer from the CLI for the latest specifications.

Sources (Primary Information)

• Let Claude use your computer from the CLI (Claude Code Official Documentation)
• Computer Use on Desktop (Claude Code Official Documentation)
• MCP (Model Context Protocol) (Claude Code Official Documentation)
• Sandboxing (Claude Code Official Documentation)
• Computer Use Safety Guide (Anthropic Support)

Related Articles

• Introduction to Claude Code — 9 Features Anyone Can Use
• What Is Claude MCP — Full List of Capabilities and 5 Connection Steps
• Claude Code Scheduling — 4 Methods and Pitfalls
• Claude Desktop App — How to Use 5 Features and Common Pitfalls
• Claude Code on Mac — brew + Apple Silicon + Notification Settings