Custom Computer Use Driver

You have your own machine: a VM, a remote desktop, or a device in a lab. No driver package controls it. Write a ComputerDriver, and computer use runs on it with no other change.

Write the driver

A driver is an object with a screen size and a small set of methods. This example sends each method to an HTTP agent that runs on the machine.

// app/my-driver.ts
import type { ComputerDriver } from '@tanstack/ai-computer-use'

export function createMyDriver(base: string) {
  async function send(path: string, body: object) {
    const response = await fetch(`${base}${path}`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(body),
    })
    if (!response.ok) throw new Error(`${path} failed: ${response.status}`)
  }

  const driver: ComputerDriver = {
    display: { width: 1920, height: 1080 },
    environment: 'windows',
    async screenshot() {
      const response = await fetch(`${base}/screenshot`)
      if (!response.ok) throw new Error(`/screenshot failed: ${response.status}`)
      return new Uint8Array(await response.arrayBuffer())
    },
    click: (x, y, { button, count }) => send('/click', { x, y, button, count }),
    move: (x, y) => send('/move', { x, y }),
    mouseDown: (button) => send('/mouse-down', { button }),
    mouseUp: (button) => send('/mouse-up', { button }),
    scroll: (x, y, deltaX, deltaY) => send('/scroll', { x, y, deltaX, deltaY }),
    keyDown: (key) => send('/key-down', { key }),
    keyUp: (key) => send('/key-up', { key }),
    press: (keys) => send('/press', { keys }),
    type: (text) => send('/type', { text }),
  }
  return driver
}

Then give it to createComputerUse:

import { createComputerUse } from '@tanstack/ai-computer-use'
import type { ComputerDriver } from '@tanstack/ai-computer-use'

// From app/my-driver.ts above.
declare function createMyDriver(base: string): ComputerDriver

const computerUse = createComputerUse({
  driver: createMyDriver('http://localhost:8000'),
})

A driver is one machine. Every thread shares it, so it serves one conversation at a time. createComputerUse never closes it. To give each thread its own machine, write a driver factory.

The contract

Coordinates are in pixels, in display space. Key names are KeyboardEvent.key names, for example 'Enter', 'Control', or 'a'.

Member	What it does
display	The screen size in pixels. Screenshots have this size.
environment	'browser', 'linux', 'mac', or 'windows'.
screenshot()	Returns PNG bytes as a Uint8Array.
click(x, y, { button, count })	Clicks count times. button is left, right, middle, back, or forward.
move(x, y)	Moves the mouse.
mouseDown(button), mouseUp(button)	Presses or releases a button at the mouse position.
scroll(x, y, deltaX, deltaY)	Scrolls at the point. The deltas are in pixels.
keyDown(key), keyUp(key)	Presses or releases one key.
press(keys)	Presses one combo, like ['Control', 's'].
type(text)	Types text as a user does.

Add the optional methods

Four methods are optional:

zoom(region): returns PNG bytes of one region at full size. The region has x, y, width, and height. computer_toolset_20260801 and computer_20251124 turn on zoom when the driver has this method. computer_20250124 does not.
cursorPosition(): returns the mouse position as { x, y }. Only computer_toolset_20260801 turns on cursor_position, and only when the driver has this method.
navigate(url): opens the URL. computerUse.tools then has the navigate tool. The tool accepts only http: and https: URLs.
close(): frees the machine. createComputerUse calls it only on a driver that a factory made.

Add this function to app/my-driver.ts. It narrows the JSON from your agent before it returns it:

export async function readCursorPosition(base: string) {
  const response = await fetch(`${base}/cursor`)
  const data: unknown = await response.json()
  const isPoint =
    typeof data === 'object' &&
    data !== null &&
    'x' in data &&
    'y' in data &&
    typeof data.x === 'number' &&
    typeof data.y === 'number'
  if (!isPoint) throw new Error('The agent sent no cursor position.')
  return { x: data.x, y: data.y }
}

Then add cursorPosition: () => readCursorPosition(base) to driver in createMyDriver.

One machine for each thread

One shared machine serves one conversation at a time. To give each thread its own machine, write a ComputerDriverFactory. createComputerUse calls create on the first action of each thread.

This example asks a pool service for one VM for each thread, and stops the VM when the thread closes:

// app/my-machines.ts
import type {
  ComputerDriver,
  ComputerDriverFactory,
} from '@tanstack/ai-computer-use'

// From app/my-driver.ts above.
declare function createMyDriver(base: string): ComputerDriver

const pool = 'http://localhost:9000'

export const myMachines: ComputerDriverFactory = {
  display: { width: 1920, height: 1080 },
  environment: 'windows',
  capabilities: { zoom: false, cursorPosition: false, navigate: false },
  async create({ threadId }) {
    const machine = `${pool}/machines/${encodeURIComponent(threadId)}`
    const response = await fetch(machine, { method: 'PUT' })
    if (!response.ok) throw new Error(`Start failed: ${response.status}`)
    return {
      ...createMyDriver(`${machine}/agent`),
      close: async () => {
        await fetch(machine, { method: 'DELETE' })
      },
    }
  },
}

Then give the factory to createComputerUse:

import { createComputerUse } from '@tanstack/ai-computer-use'
import type { ComputerDriverFactory } from '@tanstack/ai-computer-use'

// From app/my-machines.ts above.
declare const myMachines: ComputerDriverFactory

const computerUse = createComputerUse({ driver: myMachines })

The factory has these members:

Member	What it does
display	The screen size of every machine. Each driver from create has this display.
environment	The kind of every machine.
capabilities	{ zoom, cursorPosition, navigate }: the optional methods that every driver from create has. The tool is built before any machine exists, so the factory tells it.
create({ threadId })	Makes the driver of one thread. If it throws, the model gets the error. The next model turn calls create again.
close()	Optional. Frees what the threads share. computerUse.close() calls it after it closes every thread.

When a thread closes or goes idle, createComputerUse calls close() on the driver of that thread.

What the core builds for you

Your driver does only the small steps. The core builds the rest from them:

Drags: move, mouseDown, more move calls, then mouseUp.
Modifier clicks and scrolls: keyDown for each modifier, the action, then keyUp in reverse order.
hold_key and wait: the core holds the keys or waits, then continues.
Key names: the core changes OpenAI and Anthropic key names to KeyboardEvent.key names.
Screenshots: after a normal action, the core waits screenshotDelay ms and calls screenshot(). An OpenAI batch waits once, after the last action, then takes one screenshot. zoom and cursor_position return their own result. The core does not call screenshot() for those two.

Try it

Start your agent on the machine, and send a task in the chat. Your agent gets the first /screenshot request, then the clicks and the key presses. The model now uses your own machine.