Computer-Use Agents: Browser Automation, Desktop Control, Security, and Evaluation

What computer-use agents actually do

A computer-use agent observes a screen or browser state, decides what to do, and emits actions such as click, type, scroll, drag, open URL, or press key. This is useful when an application has no API or when the workflow is inherently visual.

It is also brittle. UI changes, popups, authentication prompts, slow pages, and hidden state can derail an agent that looked impressive in a demo.

• -Best: internal admin tasks, QA flows, form filling, repetitive browser work, supervised back-office operations.
• -Risky: payments, account deletion, legal filing, medical systems, privileged infrastructure consoles.
• -Better with APIs: any workflow where a stable API can replace UI control.

Architecture

A production computer-use system should run in an isolated browser or desktop session with scoped credentials. The model should not receive raw secrets, and the action executor should enforce policy even if the model asks for something unsafe.

type UiAction =
  | { type: "click"; selector?: string; x?: number; y?: number }
  | { type: "type"; text: string }
  | { type: "navigate"; url: string }
  | { type: "submit" };

function requiresApproval(action: UiAction, pageUrl: string) {
  const sensitive = ["billing", "delete", "settings", "admin", "deploy"];
  return action.type === "submit" && sensitive.some((word) => pageUrl.includes(word));
}

Observation design

Screenshots provide visual grounding, but they should be supplemented with accessibility trees, DOM snippets, URL, focused element, visible text, and known app state. This reduces hallucinated clicks and makes eval failures easier to diagnose.

• -Screenshot: layout, visual state, charts, modals.
• -Accessibility tree: labels, roles, buttons, inputs.
• -DOM subset: stable selectors and form values.
• -Tool state: current URL, cookies allowed, downloads, upload permissions.
• -Memory: task goal, completed steps, pending approvals.

Security controls

Treat a computer-use agent like an intern with a keyboard inside a sandbox. It needs task-specific access, a visible audit trail, and explicit approval before irreversible work.

• -Run in disposable browser profiles or VMs.
• -Block clipboard access unless needed.
• -Redact passwords, tokens, recovery codes, and payment details from model-visible context.
• -Require approval for submit, purchase, send, delete, deploy, invite, and permission changes.
• -Record video, action logs, model reasoning summary, and final artifacts.

Evaluation

Use public benchmarks to understand capability, but build private evals from your target apps. The key metrics are task completion, number of unsafe attempted actions, recovery from UI surprises, and human intervention rate.

• -Task success: did the expected final state occur?
• -Path quality: did the agent take a reasonable number of actions?
• -Safety: did it attempt disallowed actions?
• -Robustness: popups, slow pages, changed labels, partial failures.
• -Auditability: can a reviewer understand exactly what happened?