What we guarantee
Test automation is easy to describe and hard to verify. These are the claims we make about how AtwalLabs behaves, and for each one, the mechanism that makes it true and the test that keeps it true.
Where a guarantee has a limit, the limit is written next to it rather than left for you to find. A claim we cannot point a test at does not appear on this page.
No AI in the execution path.
Steps are translated and elements are found by deterministic code. Nothing about your application, your test data or your pages is sent to a language model during a run.
Verified by walking the import graph from every module that runs during an execution (executeActions, translateStepsToActions, componentKB, pageScanner, visionLocator, appKnowledge): nothing reachable imports an LLM SDK. Element location uses a deterministic DOM scorer; the page classifier and defect analyser are disabled stubs. With the optional legacy translation service enabled, step text is sent to that externally hosted translator; see 06.
Generation is checked for reproducibility.
Every flow edit is compiled twice and the two results compared. If they differ, nothing is saved and the difference is reported.
Checked, not assumed — and the wording is deliberate. The check covers the steps the compiler produces. It does not cover the translator that pre-flights them, whose output is cached by content, so a second call cannot disagree with the first.
Every step that locates an element records how it found it.
The strategy, selector and confidence that found each element are stored with the run and printed in the report. A step that fails records what it tried and why it stopped.
Fills, clicks and their conditional forms carry this record. Steps that locate nothing — opening a URL, waiting, asserting on the page, taking a screenshot — have no element to attribute and carry none. The vocabulary is a closed set, so a resolution is never described in free text, and runs recorded before this feature existed show "not recorded" rather than a guess.
We never invent a confidence.
A score is shown only where the resolution strategy actually produced one.
Strategies that carry no score record null and display as a dash. Where a strategy scored some steps and not others, the report says so rather than averaging the ones it has.
Unchanged suites are recognised as unchanged.
Regenerating a suite that has not changed produces the same fingerprint. A change is shown as a diff against the previous generation.
The fingerprint covers the tests: the application, the knowledge base, the mode and every case and step in order. It deliberately excludes the requirement title, so renaming a requirement is not reported as a change to its tests. A previous generation is matched by the saved flow it came from, or by title; the first generation of a flow has nothing to compare against and says so.
A step that cannot find its element fails.
If the control a step names is not on the page, the step fails. It is never reported as passing against something else.
A guard on text that is not present no longer fires either — that covers all six conditional forms, two of which were wrong in opposite directions before: an assertion that could never fail, and a failure step that always fired.
Nothing leaves the deployment by default.
Your tests, knowledge base, run artefacts and reports are written to the deployment's own disk and database — yours if you self-host, ours in AWS us-east-2 if you use the hosted service — and no integration that sends data to a third party is on unless it is switched on.
Three optional integrations send data to a third party and all three are off unless switched on: the legacy translation service (USE_SMART_API=1 — posts the test title and step text as written, and the address under test; ONE switch governs all four of its call sites, which a key on its own no longer does), the OpenAI-assisted generator (ENABLE_OPENAI_TEST_GENERATION plus a key — sends requirement text), and Browserbase (BROWSERBASE_API_KEY plus a project id — runs the browser, and therefore the pages under test, remotely). Two things are on: transactional email goes through the provider configured for the deployment, and the public marketing pages carry Google Analytics — the signed-in product does not.
Why this page exists
Every one of these started as a sentence someone wrote in a document, and a sentence in a document has no way of being wrong out loud. So each is now a row of data with a test file attached, and a build that cannot find the test fails. That is a lower bar than "correct", and a much higher one than "written down".
If you are evaluating us against a tool that generates and locates with a language model, the difference these describe is the one worth testing yourself: run the same suite twice and compare. See how we compare.