1. Computer-use and browser agents

A computer-use agent (a browser agent when the screen is a web page) is an LLM agent that operates software through its screen. Instead of calling an API, it looks at a screenshot or the page's element tree, then clicks, types, and scrolls the way a person would. It runs the basic loop from Agents with different tools. Each step, it observes the screen, picks the exact element to act on, performs one UI action, and checks that the screen changed as expected before choosing the next step.

An accounts-payable team at a mid-size distributor enters about 400 supplier invoices a day into a legacy supplier portal with no API: open the purchase order, fill the invoice number, amount, and date, attach the invoice PDF (Portable Document Format) file, press Submit. That is where an agent earns its place.

An agent is the wrong choice when an API exists for the action. Call that API as a normal tool instead. It is also a poor choice when a stable page sees the same flow hundreds of times a day, because a recorded script does that work in seconds.

The interview question behind this tutorial is "When would you drive a graphical user interface (GUI) instead of calling an API, and how do you make that reliable?" Companies that build browser or desktop automation agents probe it for a whole round. Elsewhere it is a few minutes inside an agent design round, usually right after you say "this system has no API."

A mid-level candidate asks whether an API exists, names screenshots vs the element tree, and says the agent must check the screen after acting. A senior candidate designs that check, explains how the agent clicks the wrong field and how to prevent it, prices a step in seconds and tokens, and knows which parts of the flow belong in a script. This tutorial builds the senior version on the invoice portal.