Vercel's agent-desktop: who is responsible for the session?

Version v1.1.0 of agent-desktop, published on November 3, changed how elements in applications are identified. The company delegating the task must assign responsibility for identity, access and sensitive actions in the session. Identification is not authorization. The repository has existed since February; newsletters featured the tool on November 2. Who is responsible for the session?

What happened with agent-desktop?

Vercel Labs released an update to a command-line tool that lets AI agents operate applications through the operating system's accessibility tree. Version v1.1.0 is not the initial launch: the repository API records its creation on February 19, 2026.

AlphaSignal and The Code featured the tool on November 2. The following day, v1.1.0 changed how refs, references to elements observed in a window, are reidentified.

The repository describes a CLI with an Apache-2.0 license and support for macOS, Windows and Linux, with differences across platforms. It is not an AI model: it is a tool an agent can call. Identification is not authorization.

Table: element identification in agent-desktop before and in v1.1.0. agent-desktop is a CLI that operates applications through the accessibility tree; a ref is a reference to an observed element.

The documented mechanisms do not demonstrate a production error rate. A ref identifies a target, not the purpose of the task.

Why does the desktop session matter to agent operators?

The graphical session determines part of what a desktop AI agent can observe and act on. Available applications, signed-in accounts and system permissions form a concrete access surface. The session is not the only boundary: application permissions and other access paths matter too.

The accessibility tree exposes representations of interface elements, such as buttons, fields and their names, subject to application support and environmental permissions. An agent authorized to consult a record is not thereby authorized to change its data. The inventory must include authenticated accounts: reach depends on each application's grants, not just on the task.

The README lists 60 command names, of which 56 are operational. The other four refuse execution until a process can release a held key or button. The count describes the documented surface. It does not measure risk or adoption.

A snapshot can contain interface names and values; clipboard contents, screenshots and traces can contain private data. Retaining these artifacts requires control over who can read them after execution.

This complements AI agent permissions, access and traceability, which examines identity and traceability. In the graphical session, windows and credentials are within the agent's reach; the party granting access remains responsible for supervision.

How do snapshot and ref help identify the target?

The documented flow starts with a snapshot of the interface and returns refs bound to that snapshot. The agent uses a ref to request an action and observes the state again. This helps identify the target, but does not decide whether the action is permitted or guarantee that the result persisted.

A qualified ref includes the snapshot identifier and the element identifier, such as @s8f3k2p9:e8. The snapshot identifier binds the target to that snapshot. A ref without a snapshot identifier returns INVALID_ARGS. An identified snapshot that no longer exists or has expired returns SNAPSHOT_NOT_FOUND.

These are different failures: an invalid argument or an unavailable snapshot. In both cases, the operator should observe again and select the target.

A ref must be qualified.

The v1.1.0 JSON contract also describes STALE_REF, when the ref cannot be reidentified, and AMBIGUOUS_TARGET, when identity is not sufficiently established. Refusal does not resolve the ambiguity on its own: identical elements can remain indistinguishable after another snapshot.

Snapshot completeness matters. The data.complete field indicates whether the presented tree is complete; data.identity_unproven can indicate that the full read needed to prove identity was missing. On Windows and Linux, and on macOS when there is no element address, the absence of that proof leads to refusal of the ref.

A disposition.retry value of safe, shown as retry: safe, means the action was not delivered. It does not mean repeating the same ref will fix the problem. If delivery is uncertain, an automatic retry can duplicate a change; the responsible person must check the state before deciding. Even ok can indicate delivery without verification. The interface shows the immediate result, but does not prove persistence in the application; a check appropriate to the system is required, especially when the operation changes business records.

How does this compare with an API or a pixel-based interface?

An API can restrict access by resource and operation if its scopes and effective grants are well defined. The accessibility tree offers another path: it interacts with elements exposed in the session. A pixel-based approach observes visual appearance. None of the three removes the need for authorization and verification.

If an API grants read-only access to the necessary resource, it can constrain the task better than a session with several authenticated applications. If the operation requires a graphical interface, the company must control the session and test its access. There is no basis for a universal accuracy ranking.

Least-privilege access for AI agents examines scopes and write authorization. Does the session account provide paths outside the required application? A restricted API does not, by itself, fix an excessively privileged graphical session.

In agent-desktop, ref-based actions use the accessibility API without moving the pointer or taking focus by default. That does not isolate the environment. AI agent isolation: a network claim is not enough examines isolation; the graphical session is part of that analysis, not a substitute guarantee.

Which controls need to exist before execution?

For agent-desktop, the company must define who creates the desktop session, which applications and identities it contains, who approves actions and who is responsible for the records. These controls are architecture and operating decisions, not proven security capabilities of the CLI. The boundary needs testing before write access is granted.

1. Separate the session from personal work. Use a session dedicated to the task and inventory the applications, windows, credentials and access paths available. Check operating system and application permissions. A dedicated session reduces the mixing of contexts, but does not create isolation on its own.

2. Grant the least privilege necessary. Associate the session with a specific identity whose read and write access is limited to authorized work. Test whether the account can reach other data or operations through alternative paths and remove what is unnecessary.

3. Protect records and evidence. The person responsible for the operation should record the identity, target, requested action, result and delivery signals, restricting access to the trail to protect any data it contains. Set limited retention for snapshots, screenshots and traces, because they can include private data. Do not retain full screen contents indefinitely.

4. Tie approval to higher-impact actions. Before a destructive or hard-to-reverse action, present the target and intended effect to an accountable person. Approval must correspond to the operation; general consent to use the agent does not replace that decision.

5. Test refusal, interruption and recovery. Check behavior with a missing ref, an ambiguous target and an uncertain delivery result. When interrupting a run, confirm the state in the application before resuming. If the persisted effect cannot be verified, the task needs review, not an automatic sequence of further retries.

agent-desktop alone does not prove isolation, human approval, a tamper-proof audit trail or exfiltration prevention. The buyer should demand evidence of these controls and establish who can suspend execution when the session exceeds the necessary access or a change cannot be confirmed sufficiently. The same standard applies to AI agents in B2B: the complete guide for 2026.

Frequently asked questions

agent-desktop identifies elements and forwards actions; the company defines authorization for the task in the graphical session. Security in real systems depends on the session and the environment's other controls. A technical success response does not replace checking the result in the application.

Is agent-desktop an AI model?

No. It is a command-line tool for reading and operating applications through the accessibility tree. An agent can call it as part of its workflow, but the CLI is not itself the model that interprets the request or decides the task's objective.

Does headless mean the session is isolated?

No. Headless describes ref-based actions that do not move the pointer or take focus by default. The term does not guarantee account separation, network blocking or application restrictions. Isolation must be tested. The absence of pointer movement does not prove that separation.

Does a valid ref authorize a click?

No. A valid ref helps identify the element targeted by the action. Permission depends on the task, the identity and company policies. A technically possible action can still be prohibited. Identification does not replace write authorization.

When can an action be retried?

The contract marks retry: safe when the action was not delivered. Even then, the cause must be addressed: repeating the same ref does not resolve indistinguishable elements. If delivery is uncertain or unsafe to retry, the operator should check the state and refer the decision to the responsible person.

References and further reading

The permissions documentation details the requirements for each operating system, while the project FAQ helps identify which artifacts can contain private data, without proving how a company will protect those records in practice. Version v1.1.0 is the documentary basis for the technical limits described here. Documentation is not a security certification.

Who should control the session?

The company delegating the task to agent-desktop must assign responsibility for the identity, available applications and authorization of each sensitive action in the graphical session, including the decision to suspend execution when the limits cannot be verified. When evaluating agents, including Nexforce Agents, that responsibility needs to appear in the design and tests of the operation.

Vercel's documentation does not establish an integration with Nexforce. The practical decision is to start with the least access that can be verified and test the boundary before granting write access. The right target does not authorize an action.