BAT — Browser Automation Tool
A study in making an autonomous agent reliable against an environment it does not control. BAT builds an accessibility tree of the live DOM with stable numeric references, unifies references across cross-origin iframes into one flat registry, and normalises the tree after every action so a no-op is detected by comparison rather than assumed successful. Failures escalate through a fixed ladder of recovery strategies — retry with a trusted CDP click, then suggest refresh or navigation, then stop and hand control back.
Closed-loop agent with cross-frame reference unification, fingerprint-based staleness detection, and a graceful escalation ladder. Reads the page as structured text, not pixels. 486 automated tests gate every change.
Most browser bots are either told what to do, click by click, or they stare at screenshots and guess. BAT does neither. It reads the page the way a screen reader would, then browses and acts on it for you.
A tiny co-pilot that lives in your browser.
Open the side panel, type a goal in plain English, and BAT takes it from there. It moves the mouse, clicks through the pages, fills in the forms, and keeps going until it is done or it hits something it cannot resolve. The interesting part is not the clicking. It is knowing, after every action, whether the page actually changed.
Repetitive navigation, handled reliably.
Every one of these is the same underlying problem: drive a stateful interface you don't control, and know for certain whether each action worked.
Text, not pixels.
Instead of taking screenshots and asking a model to squint at them, BAT hands the model a clean outline of the page — every button, link and heading, labelled and numbered.
link "Pricing" #11 link "Docs" #12 button "Download report (PDF)" #13 heading "Your invoices — page 2 of 7"
The number next to each element is a name tag. The model can say "click #13" three turns later and BAT still knows which button it means, even if the page has redrawn itself in the meantime. Cheaper than screenshots, sharper than screenshots.
Two speeds — quick, or quiet-and-careful.
For simple pages, BAT just talks to the page directly — the same way JavaScript on the page would. Instant, cheap, works for the 90% case.
For pages that don't trust scripted clicks, BAT drives the browser at a lower level — real mouse movements, real keystrokes. Indistinguishable from a person, and it works even when the tab is in the background so you can keep using your computer.
Pages inside pages inside pages.
A lot of dashboards, admin portals and embedded players live inside an iframe, a page nested inside another page, often served from a different origin. Browsers deliberately stop those pages reading each other. That is correct for security and a nightmare when you need to resolve a coordinate inside the embed.
Every embedded page has its own coordinate system, starting at (0, 0) in its own corner. To click something, BAT needs coordinates for the whole window — but no page is allowed to see outside its own walls.
BAT asks each layer one small question — "where does your child live inside you?" — and adds up the answers. Nobody has to see through the whole building; they just point down to the next floor.
It notices when nothing happens.
After every action, BAT takes a tiny snapshot of the page. If the snapshot is identical to the one before, it means the last click did nothing — and instead of clicking harder, it tries a different tactic.
Two dud clicks — switch from a scripted click to a "real" one and try again.
Still stuck — try scrolling, refreshing, or going back a page.
Give up gracefully — tell you what it tried and hand the tab back.
It bends, it doesn't break.
Every piece BAT depends on has a plan B. If the AI can't handle screenshots, BAT drops them and keeps going with text only. If streaming responses aren't allowed, it falls back to a normal request. If that's blocked too, it routes through a different part of the extension. You usually don't notice — the agent just keeps working.
The "Resume where you left off?" trap.
Stateful apps often greet you with "Resume where you left off?" the moment they open. A naive agent clicks Resume, the restore silently fails, the dialog returns, and it clicks Resume again, forever. This is the general failure mode: an action that looks successful but changes nothing. BAT counts repeats. If Resume fails twice it switches to Restart automatically, and before every click it re-reads the label under the cursor so a Restart-intended click can never land on Resume. The same counting logic drives the whole escalation ladder.
Boring, on purpose.
No React, no state library — hand-written HTML/CSS/JS, wired together by Vite and @crxjs/vite-plugin, which compiles the manifest into the MV3 bundle and gives content scripts real hot reload in dev. It's a one-person project, and every choice is aimed at keeping the loop between "change something" and "see it work" as short as possible.
The engineering detail, if you want it.
For anyone who wants the engineering detail — everyone else, feel free to skip this part.
What it isn't, yet.
- Coverage stops at pure logic.486 assertions gate dedup, the plan/runner state machine, extractor safety, redaction, and the allowlist — but the agent loop itself (clicking, typing, the escalation ladder) is still validated by hand against representative sites.
- Locked to one AI provider.The plumbing assumes DeepSeek's API shape. Swapping providers is future work, not a config toggle.
- Screenshots are off entirely, not just fragile.Vision support is hard-coded off because DeepSeek's chat API doesn't accept image input at all. Canvas-rendered pages fall back to reading element geometry with JavaScript and acting on raw coordinates — not to pixels.
- Hard stop at 200 steps.A genuinely long task gets cut off alongside a genuinely stuck one, because the limit counts steps rather than progress.
Next.
- →End-to-end tests of the agent loop itself — clicking, typing, the escalation ladder — the one thing the 486 unit tests don't cover.
- →Support for more than one AI provider.
- →Image input support, once the model API accepts it — for canvas-heavy pages tree-reading can't describe.
- →A step limit that adapts to the length of the task.
- →A friendlier first-run setup for the API key.