Incomplete is a third bucket

What this is

A builder-to-builder field note on one specific wall: automated keyboard-accessibility checks throw too many false positives on modern JavaScript / utility-CSS sites. Nothing here is new — axe's incomplete bucket, the ACT rule grammar, accessibility-tree snapshots, local vision-language models. It organizes what's already known rather than claiming anything new; the contribution is the sort order, two levers you can actually ship, and an honest map of where the evidence runs out. (Engineering experience, not a compliance or legal opinion — WCAG conformance carries legal weight in some contexts; check the primary sources and your own requirements.)

How to read the tags. The evidence here is thin — vendor docs, three preprints, one practitioner benchmark — so every load-bearing claim is tagged by what backs it, and each citation is a click-to-highlight jump into the source:

• sourced — a verbatim quote from a named source backs it.
• author's read — my reasoned inference; no source benchmarks it.
• practitioner-reported — common field framing, not from a cited study.
• open gap — nobody, in these sources, has measured it.

incomplete is a third bucket

The single most useful move rests on the most solid quote. axe-core reports two channels that pipelines routinely collapse into one: confirmed violations, and incomplete (Deque also calls these “needs review”). An incomplete just means “axe-core wasn't able to tell whether something was a violation or not with 100% certainty”¹ sourced. That is undecided, not a verdict — it can hide a real violation as easily as a non-issue — and axe reserves its no-false-positives guarantee for the other bucket: “Because of axe-core's focus on zero false positives, we only include violations we're sure are actual violations”² sourced. So an incomplete is not “a false positive”; it's a question the engine is handing you.

Most harnesses assert on violations and silently drop incomplete. On a static page that's tolerable. On a JS / utility-CSS site, the checks axe can't resolve with certainty — contrast computed over a gradient or a background image, say — pile up in incomplete, and dropping them silently is how a real issue slips through and how a non-issue gets miscounted as noise author's read. Treat it as a third bucket and route it:

const { violations, incomplete } = await axe.run(target);
report.fail(violations);     // confirmed  -> block the build
triage.enqueue(incomplete);  // undecided  -> human / AI-assisted review,
                             //               never auto-fail, never auto-pass

That one split turns a wall of red into two queues with different owners — and it's the spine the rest of this note hangs on. (Whether your false positives actually concentrate in incomplete rather than violations is an open question — see the gaps below.)

Three false positives wearing the same hat

“Too many false positives on JS/Tailwind sites” is usually three different failures that look identical in a report and need opposite responses. Sort by cause first:

One concrete screen for the second row: run it twice and diff the violations. A finding present before the view settles and gone after is a strong candidate for a timing artifact rather than a real defect author's read. Treat it as a screen, not a proof: a clean re-run changes both the elapsed time and the run, so a finding that clears shows non-persistence, not timing-causation — and a settled re-run can also mask a genuinely transient violation. Neither the second nor third row is the tool being “wrong”; they're real signals about your build. (And an empty violations list was never a clean bill of health — “many accessibility problems can only be discovered through manual testing”³ sourced.)

Work backwards from examples: pin the expectation to a contract

The runnable version of “work backwards from examples” already ships. Playwright's accessibility-tree snapshots let you “assert the accessibility tree of a page against a predefined snapshot template”⁴ sourced — a declarative expectation you capture once from a known-good example and assert against thereafter. The catch is printed on the label: the docs call them “Not ideal for highly dynamic content where outputs change frequently or unpredictably.”⁵ sourced. So the boundary is sharp: declarative expectations hold on stable component contracts and degrade on exactly the dynamic, frequently-changing views where your false positives live author's read. Pin them to the components that are stable; keep the volatile views on the triage path above. The cost is honest — authoring and maintaining those contracts is real work, and it should narrow the noise by shrinking what you assert, not by making detection cleverer author's read — no source benchmarks the reduction.

If you'd rather author those expectations in a shared grammar than a bespoke format, the W3C ACT Rules Format (version 1.1, a Recommendation since February 2026) is built for it: “The test rules can be used for developing automated testing tools and manual testing methodologies”⁶ sourced. Its core is an applicability (what a rule covers), an expectation (“description of the expected characteristics of the applicable rule content”⁷), and test cases (“sample code demonstrating passed, failed, and inapplicable rule conditions”⁸) sourced — your worked examples, formalized. It's a rule-authoring grammar, not a runnable noise-killer; adopting it doesn't make a scanner smarter.

Local AI: an annotator, not a judge

For the local-model step, one practitioner benchmark is directly on point. Testing local vision-language models for alt-text, “Among local options, the Llama variants and MiniCPM-V perform best”⁹, and “Both earned a B grade: they work reliably but sometimes miss important details”¹⁰ practitioner-reported. A usable starting point for this week's work — scoped to generating a candidate description, not deciding conformance.

Why that scoping matters is a documented failure mode — and the provenance has to be exact, because it's the easiest thing here to over-read. In a 2025 preprint benchmarking cloud models as judges for PDF accessibility, Claude-3.5 in particular “Claude-3.5 occasionally hallucinated alt text descriptions for images without alt text and then marked the document as Failed based on those hallucinated descriptions”¹¹; and for undecidable cases, “all models except GPT-4-Turbo incorrectly classified documents as 'Failed', also due to predictions based on hallucinated descriptions”¹² sourced — GPT-4-Turbo was the exception there, and the study's most accurate model. Be precise about what this is: cloud frontier models, on PDFs, acting as the judge. Whether the same silent failure shows up for a local model on a live DOM is author's read, not the study's finding — but the mechanism, a model inventing the evidence and then ruling on its own invention, is the one to engineer against.

So the line that keeps it useful is narrow: let the model annotate a finding the tool already localized, require that its cited evidence actually exists in the DOM before you trust the verdict, and don't make it the detector of record. A model can read structure well — ScreenAI is “a vision-language model that specializes in UI and infographics understanding”¹³ sourced — but reading structure is extraction, not adjudication.

Why a local model can't be your detector of record — yet

There's a cited, structural reason this is more than caution. A 2026 systematic review of LLMs for web accessibility finds that “most studies apply LLMs to text-centric and structurally explicit accessibility tasks, with WCAG serving as the primary reference framework and limited consideration of cognitive accessibility guidelines (COGA)”¹⁴ sourced. Scope it precisely: that is a gap in cognitive-accessibility coverage and in the labelled-data / annotation quality behind it — not a blanket ceiling on visual or DOM-level checking. But it's the concrete reason a small local model, trained on what little labelled accessibility data exists, makes a capable assistant and a poor authority for now.

The floor you already work to

Two baselines you know cold, stated only because they bound the exercise in both directions. Automated absence is not presence: “Absence of detected errors does not indicate that a page is accessible or conformant.”¹⁵ sourced. And the ceiling is structural — per W3C's evaluation methodology, “most accessibility checks are not fully automatable, evaluation tools can significantly assist evaluators”¹⁶, and as the PDF-benchmark paper puts it, “manual testing has become essential for identifying issues that automated tools cannot detect”¹⁷ sourced. Fewer findings isn't better; it's just quieter. (Which is the whole reason you built a keyboard harness: operability — “All functionality of the content is operable through a keyboard interface without requiring specific timings for individual keystrokes”¹⁸ sourced — is a behavioral property a static scan can't confirm.)

Does any of this pay for itself — and the failure mode that bites back

Two questions a skeptical reader asks first. Does it pay? Honestly, unknown: the false-positive reduction from every lever here is unmeasured, manual testing stays essential, and maintaining declarative contracts is ongoing work. The case for these moves is that triage-routing and component contracts are cheap to try and easy to back out — not that they're proven to net positive author's read. What's the failure mode? Every move here quiets the tool, and a quieter tool raises the false-negative risk — the worse error on an accessibility surface. If the incomplete queue outgrows whoever reviews it, “route it to triage” becomes “route it to a graveyard,” and real violations sit there unread. Two guards worth budgeting for: staff triage as a real line item, and fingerprint incompletes so the same unresolved item collapses to one entry across CI runs instead of re-flooding the queue author's read. Stable fingerprints across runs are also what let you tell a genuinely new incomplete from yesterday's known one.

What nobody has measured (the most useful part)

On thin evidence, the honest gaps are worth more than confident prose. In priority order:

• FP-reduction on modern-framework sites is unmeasured. No cited source benchmarks how much any of this — incomplete-triage, declarative contracts, AI annotation — actually reduces false positives on JS / utility-CSS sites open gap. Measure it and that's a contribution.
• Where the false positives concentrate is unmeasured. Whether they sit mostly in incomplete or in violations on framework-heavy sites is, as far as these sources go, unquantified open gap.
• Scan timing vs. hydration is folklore, not data. The “re-run after the page settles” screen is widely practiced but, in these sources, unquantified open gap.
• Local-model adjudication on a live DOM is untested. The hallucinate-then-judge failure is documented for cloud models on PDFs; no cited source tests it for a local model on web components open gap.

Evidence quality & limits

• Strongest (definitional, not empirical): the axe incomplete semantics (vendor docs); the ACT and aria-snapshot grammar (W3C Recommendation + tool docs).
• Moderate: the hallucinate-then-fail failure mode — one 2025 preprint, cloud models, PDFs.
• Thin: the local-model grades (one practitioner benchmark, alt-text only); the cognitive-accessibility gap (one systematic review).
• Reasoning, not cited measurement: the timing cause (open gap) and the framework-markup cause (practitioner-reported); every false-positive-reduction benefit.
• Scoped to the sources cited, as of June 2026; none of the tooling claims were independently benchmarked for this note.