skip to content

Search
Independent build · Voice + WebMCP proof of concept Sept 2026
Shipped

VocalApply — a Voice + WebMCP proof of concept for agentic form-filling

A working proof of concept: a voice agent searches a careers board, opens a role, and fills an entire application by conversation, running on WebMCP, Gemini Live and OpenAI Realtime with no backend. It can search, fill, and correct every field. Nothing in it can press submit, by design.

React 19
TypeScript
Vite
WebMCP
Gemini Live (WebSocket)
OpenAI Realtime (WebRTC)
TailwindCSS 4
View on GitHub
# ai
# voice
# webmcp
# agents
# product-management
# product-design

Outcomes

2Tool scopesboard (6 tools) and application (9 tools) — never both registered at once
6Verification scriptsmeasured claims against a real browser, not asserted ones
0Submit calls a tool can makeprepare_submit focuses the button and returns submitted: false, always

Three pillars

01

One search, two callers

src/lib/matchOpenings.ts is the search, and it exists once. The filter bar and the find_matching_roles tool call the same function with the same argument shape, so a shortlist an agent reads aloud is reproducible by a person with the controls in front of them.

02

Fill what it knows, ask for the rest

fill_step writes every field the profile holds and reports the rest as unanswerable. ask_for_field opens that step and highlights the one question left, so you see what it means before it asks. Neither tool can write the declaration checkbox — a tool that could would tick 'I have read this application' on your behalf.

03

No key in the bundle, and it's measured, not assumed

OpenAI: the server mints a short-lived token, then the browser talks to OpenAI directly — no audio crosses the server. Gemini has no equivalent path on its Live socket; every combination of token, key and API version was tried and closes 1008 'unregistered callers', so Gemini asks for your own key rather than pretending otherwise.

VocalApply is a voice agent that operates a real web app end to end: it searches a careers board, opens a role, and fills an eighteen-field form by conversation, running on WebMCP, Gemini Live and OpenAI Realtime with no backend holding your data. It’s also built to know exactly where its authority ends. Careers pages are the most confidently automatable UI on the web, and the most dangerous one to actually automate: a form an agent can fill is a form an agent can submit, and a submitted application with a fabricated answer is a lie with your name on it. That’s the constraint this build exists to prove out.

WebMCP is new enough that almost no one has built against it yet. It’s document.modelContext, a proposed browser API that lets a page register tools an AI agent can call directly, without scraping the DOM or guessing from the accessibility tree. Building on it surfaced a real compatibility problem: ChatGPT’s own in-browser agent already ships a non-configurable stub of document.modelContext, and calling Object.defineProperty over it throws instead of installing. One of the six verification scripts in the repo exists specifically to handle that: adopt whatever the browser already put there instead of assuming a clean slate.

Try it

Live: vocalapply.trustvalidated.com · Demo video: YouTube · Code: github.com/hoshank/VocalApply

Open the live site, press Start with shared credit in the voice dock (no key ever touches the page), and say:

  • “What have you got open?” Searches the board, sets the filter bar, reads the top match.
  • “Apply to that one.” Opens the role and its form.
  • “Fill it in.” Fills everything the profile holds, then asks about the one thing it can’t answer.
  • “Submit it.” It will not, and it says why.

The rule the whole thing is built to keep

No tool submits, and no spoken instruction can. prepare_submit calls focus() on the button and returns submitted: false. Voice makes “just say yes and send it” the obvious next feature to build. This demo exists specifically to refuse it.

Measured against a real browser, not asserted: 18 fields filled, the declaration checkbox left unticked, submitted: false. There’s a second, narrower demo behind the same build, at ?tdd=1, that shows a WebMCP tool call surviving a restyle and a field-splitting refactor that break a recorded Playwright selector and a data-testid in turn. The claim on screen is narrow on purpose: it fixes tests that break when markup moves, nothing about multi-tab flows or cross-origin logins, and no invented percentage anywhere on the page.

What’s actually hard about this

Two tool scopes that never overlap. Browsing the board while carrying fill_step in the registry would describe a form that isn’t on screen. list_open_roles, find_matching_roles, next_role, open_role and two more live in the board scope; fill_step, correct_field, ask_for_field, prepare_submit and four more live in the application scope. A self-check asserts the live registry is exactly one of the two sets, at boot, so a leak fails loudly instead of shipping quietly.

Two passes, then it asks you fill_step writes what the profile knows. What it can't answer becomes a question, not a guess. Grey monospace text is the field count at each step. Scroll slowly, this plays once. Applicant profile Whatever this browser already knows about you resume, prior answers fill_step runs Writes every field the profile can answer 17 of 18 fields, one pass Can't answer this one "Open to relocating?" a real choice, not a lookup relocation: unanswerable ask_for_field opens it Highlights the step and the field, then waits you see the question first correct_field writes it Your answer lands, the highlight clears 18 of 18, none guessed what the profile already holds a question the profile can't answer for you a tool call, in order Neither writing tool can tick the declaration checkbox — that answer is never filled for you.
Two tool calls do the writing. Neither one can tick the declaration checkbox; that answer is never filled for you. (animates as you scroll to it) PNG ↓

Filters are hard rules, matching is a soft rank. Work mode, contract, location, and salary each remove a role from the board, and the tool reports how many each filter took out, so a search that returns nothing can be widened instead of abandoned. Profile matching, by contrast, only orders the results; it never hides a role a person might still want to apply for.

Credentials that are measured, not hoped. The write-up in the repo documents an actual dead end: every combination of Gemini’s auth_tokens (bound, unbound, v1beta, v1alpha) completes the WebSocket handshake and then closes the moment a real message is sent. That’s a test result, not a guess about Gemini’s API, which is why the app asks for a Gemini key instead of quietly building a relay that would route your voice through a shared server.

Why this is here

This project is the sharpest artifact I have for what “agentic product design” actually means: not wiring a model to a tool schema, but deciding, in writing, what an agent is never allowed to do, then proving with a script that the code keeps that promise.

Related reading