BAT — Browser Automation Tool
A study in making an autonomous agent reliable against an environment it does not control. BAT builds an accessibility tree of the live DOM with stable numeric references, unifies references across cross-origin iframes into one flat registry, and normalises the tree after every action so a no-op is detected by comparison rather than assumed successful. Failures escalate through a fixed ladder of recovery strategies — retry with a trusted CDP click, then suggest refresh or navigation, then stop and hand control back. Beyond single-page browsing it now does structured bulk collection: the model synthesises a page-reading extractor once per site, caches it in IndexedDB keyed by URL pattern, then replays it across hundreds of pages with zero further model calls — every replay validated before it is trusted. Where a company exposes a public ATS board (Greenhouse, Lever, Ashby, Workable), it skips the browser entirely and pulls job listings straight from the JSON API.
Closed-loop agent with cross-frame reference unification, fingerprint-based staleness detection, and a graceful escalation ladder. Reads the page as structured text, not pixels — and replays cached extractors for bulk data collection. 502 automated tests gate every change.
Most browser bots are either told what to do, click by click, or they stare at screenshots and guess. BAT does neither. It reads the page the way a screen reader would, then browses and acts on it for you.
A tiny co-pilot that lives in your browser.
Open the side panel, type a goal in plain English, and BAT takes it from there. It moves the mouse, clicks through the pages, fills in the forms, and keeps going until it is done or it hits something it cannot resolve. The interesting part is not the clicking. It is knowing, after every action, whether the page actually changed.
Repetitive navigation, handled reliably.
Every one of these is the same underlying problem: drive a stateful interface you don't control, and know for certain whether each action worked.
Text, not pixels.
Instead of taking screenshots and asking a model to squint at them, BAT hands the model a clean outline of the page — every button, link and heading, labelled and numbered.
link "Pricing" #11 link "Docs" #12 button "Download report (PDF)" #13 heading "Your invoices — page 2 of 7"
The number next to each element is a name tag. The model can say "click #13" three turns later and BAT still knows which button it means, even if the page has redrawn itself in the meantime. Cheaper than screenshots, sharper than screenshots.
Two speeds — quick, or quiet-and-careful.
For simple pages, BAT just talks to the page directly — the same way JavaScript on the page would. Instant, cheap, works for the 90% case.
For pages that don't trust scripted clicks, BAT drives the browser at a lower level — real mouse movements, real keystrokes. Indistinguishable from a person, and it works even when the tab is in the background so you can keep using your computer.
Pages inside pages inside pages.
A lot of dashboards, admin portals and embedded players live inside an iframe, a page nested inside another page, often served from a different origin. Browsers deliberately stop those pages reading each other. That is correct for security and a nightmare when you need to resolve a coordinate inside the embed.
Every embedded page has its own coordinate system, starting at (0, 0) in its own corner. To click something, BAT needs coordinates for the whole window — but no page is allowed to see outside its own walls.
BAT asks each layer one small question — "where does your child live inside you?" — and adds up the answers. Nobody has to see through the whole building; they just point down to the next floor.
It notices when nothing happens.
After every action, BAT takes a tiny snapshot of the page. If the snapshot is identical to the one before, it means the last click did nothing — and instead of clicking harder, it tries a different tactic.
Two dud clicks — switch from a scripted click to a "real" one and try again.
Still stuck — try scrolling, refreshing, or going back a page.
Give up gracefully — tell you what it tried and hand the tab back.
Read one page with a model. Read the next four hundred without one.
Browsing one page is the easy half. The expensive half is doing it four hundred times. BAT's answer is to spend a model call once per site, turn what it learned into a small reusable reader, and then never ask the model again — while checking, every single page, that the reader is still telling the truth.
The model authors a page-specific extractor function, which is cached in IndexedDB keyed by URL pattern with full version history, then replayed on every later page of that site with zero model calls. Each replay is validated before it's trusted: schema fingerprint, empty-required-field checks, row-count-collapse detection, empty-page detection, and an explicit report when the 2,000-rows-per-page cap truncates. A failing extractor gets one retry, then the site halts — collecting nothing beats collecting garbage.
Greenhouse, Lever, Ashby and Workable publish job boards as public JSON. Where one exists, BAT pulls the whole board in a single HTTP call — no tab, no tree, no extractor. Slug discovery verifies the board's own company name before trusting a guessed slug. Measured, not assumed: of 35 Irish tech companies probed, only 2 had a verifiable public ATS board. Most real collection work still goes through the browser path.
Rows land in a deduplicating IndexedDB store keyed on a normalized composite of fields — case, punctuation and whitespace insensitive, but tight enough that "C++ Developer" and "C Developer" still don't collide. A duplicate merges its source into the first row it matched; only novel rows reach the output file. The file is always a regenerable projection of the store, never the record itself.
Collection runs in the service worker as a resumable state machine, checkpointed after every single page. It survives the panel being closed, the service worker being evicted (a 30-second chrome.alarms watchdog revives it) and a full browser restart via onStartup recovery. CAPTCHAs and login walls are detected, never solved — the run parks as AWAITING_HUMAN and resumes at the exact interrupted page once a person clears it.
Every transition is written to disk before it takes effect, so the run can be picked up again from the exact page it stopped on — after a panel close, a service-worker eviction, or a full browser restart.
It bends, it doesn't break.
Every piece BAT depends on has a plan B. If the AI can't handle screenshots, BAT drops them and keeps going with text only. If streaming responses aren't allowed, it falls back to a normal request. If that's blocked too, it routes through a different part of the extension. You usually don't notice — the agent just keeps working.
The "Resume where you left off?" trap.
Stateful apps often greet you with "Resume where you left off?" the moment they open. A naive agent clicks Resume, the restore silently fails, the dialog returns, and it clicks Resume again, forever. This is the general failure mode: an action that looks successful but changes nothing. BAT counts repeats. If Resume fails twice it switches to Restart automatically, and before every click it re-reads the label under the cursor so a Restart-intended click can never land on Resume. The same counting logic drives the whole escalation ladder.
Boring, on purpose.
No React, no state library — hand-written HTML/CSS/JS, wired together by Vite and @crxjs/vite-plugin, which compiles the manifest into the MV3 bundle and gives content scripts real hot reload in dev. It's a one-person project, and every choice is aimed at keeping the loop between "change something" and "see it work" as short as possible.
The engineering detail, if you want it.
For anyone who wants the engineering detail — everyone else, feel free to skip this part.
What it isn't, yet.
- Coverage stops at pure logic.502 assertions gate dedup, the plan/runner state machine, extractor safety, redaction, and the allowlist — but the agent loop itself (clicking, typing, the escalation ladder) is still validated by hand against representative sites.
- Locked to one AI provider.The plumbing assumes DeepSeek's API shape. Swapping providers is future work, not a config toggle.
- Screenshots are off entirely, not just fragile.Vision support is hard-coded off because DeepSeek's chat API doesn't accept image input at all. Canvas-rendered pages fall back to reading element geometry with JavaScript and acting on raw coordinates — not to pixels.
- Hard stop at 200 steps.A genuinely long task gets cut off alongside a genuinely stuck one, because the limit counts steps rather than progress.
Next.
- →End-to-end tests of the agent loop itself — clicking, typing, the escalation ladder — the one thing the 502 unit tests don't cover.
- →Support for more than one AI provider.
- →Image input support, once the model API accepts it — for canvas-heavy pages tree-reading can't describe.
- →A step limit that adapts to the length of the task.
- →A friendlier first-run setup for the API key.