OculiX | Blog

AI agentic vs deterministic visual automation in 2026: what's actually true

Thu, 18 Jun 2026 08:00:00 GMT

In April 2026, AskUI published a comparison piece framing the choice in visual automation as if it were already decided. Deterministic, image-based automation tools are presented as a legacy paradigm — useful for nostalgia, limited by screen resolution, blocked on headless execution, helpless on mobile, doomed to be replaced by LLM-piloted agents that “understand” the interface rather than match pixels against it.

I read AskUI’s article carefully. It is well-written, and the SEO will travel. It is also, on most of its load-bearing claims about OculiX, factually wrong — by several major and minor releases.

The intent of this post is not to attack a competitor. The intent is to factualize, calmly and in writing, what OculiX 3.0.4 actually does in 2026, so that a buyer reading both pieces side by side can make a decision on observable facts rather than on the carefully cropped excerpt AskUI chose to publish.

The two paradigms, named honestly

Before going into claim-by-claim correction, let me name the two paradigms fairly, because they are not the same shape and they do not serve the same constraint.

Deterministic visual automation

A script written in code. A pattern stored as an image. A matcher that decides, by pixel comparison, whether the pattern is on screen. Same input, same output, every time. No model, no token, no cloud round-trip in the hot path. Auditable end to end.

LLM-piloted agentic automation

A natural-language instruction. A multimodal model that interprets the screen and decides what to click. Same input, plausibly similar output, never bit-for-bit reproducible. Behavior changes when the model version changes. Token-billed by the round trip. Auditability is a research topic.

These two paradigms are not interchangeable in any professional QA practice. The deterministic paradigm produces reproducible artifacts on demand: production regression, compliance-grade audit trails, air-gapped perimeters, every test that must produce the same artifact today, tomorrow, and in three years on a different machine. The agentic paradigm, by construction, cannot — and a QA team optimizing for trust, audit and reproducibility cannot rationally adopt it as its primary engine. This is the position I have publicly defended on LLM non-determinism applied to automation: selling a non-reproducible autonomous execution as a test product is a professional fault, not an acceptable trade-off.

AskUI’s piece nonetheless claims that the agentic paradigm is the inevitable successor to the deterministic one, and bolsters that framing with claims about the deterministic side that are either out of date by two minor releases, true of one tool but extended to another by amalgam, or simply incorrect. It is the combination of the two — a wrong strategic framing and a wrong technical assessment — that makes the present response necessary.

Let me go through each claim.

Claim by claim

Claim 1 — “OculiX 3.0.1, a development build”

AskUI describes OculiX as “a development build called OculiX 3.0.1 […] also available with VNC support, Android ADB control, PaddleOCR integration, and additional scripting languages”.

What is actually shipping in June 2026:

Item	Reality
Current stable	OculiX 3.0.4, released and downloaded by close to a hundred organizations in production
Next major	OculiX 4.0, scoped, in active development, targeted for mid-2026
Deployments	Fortune 500, APAC enterprises, defense sector — observable through Scarf usage telemetry
OCR engine	Tesseract embedded statically via Legerix, self-contained, no PaddleOCR dependency in stable

Calling OculiX a “development build” in AskUI’s comparison written for a 2026 audience is not a small inaccuracy. It tells the reader that the alternative they are evaluating is unstable, when the opposite is true. The choice of version number “3.0.1” in AskUI’s piece appears to refer to a release from a previous quarter; their comparison was published months later, after several stable releases had shipped.

Claim 2 — “Headless execution: not supported”

AskUI asserts that headless execution is “not supported” and that “a real, unlocked screen is required”.

This is true of SikuliX run locally, on the laptop in front of a developer. It is not true of OculiX in the configuration most production teams actually deploy.

OculiX ships a remote VNC architecture, documented and supported, where:

The OculiX runner executes on a dedicated VM
The script connects to the VM through SSH, with the RFB protocol tunneled inside the SSH session
The VM session is invisible to the developer’s machine while the test runs
The developer’s machine remains fully usable for other work in parallel

This is the production deployment used by the majority of OculiX deployments at scale. It is not headless in the strict sense of “no framebuffer ever rendered” — that would be physically incompatible with pixel matching as a paradigm. It is headless in the sense the buyer actually cares about: the test does not block the operator’s screen, the test runs unattended in CI, the test does not require a human to be logged in physically in front of the test machine.

AskUI extends a true limitation of SikuliX local-mode execution to all image-based automation, which is misleading. OculiX moved past this limit several years before their comparison was written.

Claim 3 — “The machine is not usable for other user interaction while the test runs”

This is the same architectural confusion as claim 2. In a local OculiX session — same as in a local AutoIt, AutoHotKey, or pyautogui session — the mouse and keyboard belong to the running script. This is intrinsic to driving a desktop UI, regardless of vendor.

In a VNC session OculiX deployment, the test runs in a separate framebuffer on a separate VM. The developer’s mouse and keyboard are not contested. This is exactly what production teams running visual regression at scale ask for, and it is exactly what OculiX provides. AskUI frames this as a categorical limit of image-based automation; in practice, the limit was lifted years ago by the architecture of how the tool is deployed.

Claim 4 — “No execution caching capability”

OculiX 3.0.4 ships a persistent locator cache mechanism, embedded inside the PNG files used as patterns. Each pattern’s last successful match coordinates are stored as a private ancillary PNG chunk (oPLx), preserved by every standard image tool, survived by git clone, and read by OculiX on the next cold-start invocation.

The measured impact on a five-year-old i3 laptop, in a strict cold-start configuration mimicking a CI environment, is a ×6.5 speedup on the find operation: from a 502 ms baseline down to 77 ms median. On modern hardware the absolute numbers shrink but the relative speedup stays at ×6 to ×8.

This is documented in detail in my benchmark post from May 2026, with the methodology, the harness, and the raw numbers.

AskUI’s claim that OculiX has “no execution caching capability” was published after the feature shipped and after the benchmark was public. It either reflects a research process that did not look at the changelog, or a deliberate omission of an inconvenient counter-example.

Claim 5 — “No mobile support in stable version”

OculiX 3.0.4 ships first-class Android support, exposed through ADB, with device selection by serial, idempotent wake/unlock via KEYCODE_WAKEUP, and parallel multi-device farms. This has been the case for several minor releases.

The interface is documented, the code is in the public repository, and the test cases that exercise it are in the test suite. A simple git log --grep android on the public repository surfaces the recent commits introducing the per-serial device selector and the wake decoupling.

AskUI’s 2026 comparison claims OculiX has “no mobile support” — this is either months out of date or deliberately reads a single old branch. Either way, it is wrong about the stable release a buyer would actually download today.

Claim 6 — “Screenshot pattern matching breaks under resolution changes”

This claim is technically true of the naive form of template matching — the form practiced by image-based tools in 2010. It is not true of OculiX’s Finder.java implementation in 2026, which cascades five matching modes:

Mode	Strategy	Tolerance
1	Standard template match	None — fast path for exact match
2	DPI-aware rescale	Resolution and scaling differences between capture and target
3	Gaussian-blurred match	Antialiasing, subtle color variation
4	Grayscale-converted match	Color theme drift
5	Multi-scale brute force	0.5× to 2× scaling, last resort

A pattern captured on a 1920×1080 screen at 100% scaling and matched on a 2560×1440 screen at 125% scaling is the exact case mode 2 was designed to handle. The matcher does not “break”; it changes strategy and finds the pattern. This has been in the codebase for years.

Presenting the 2010 limitation as if it still defined the 2026 state of the tool is a common debate trick: pick the worst version of the position you are arguing against, and treat it as if it were the only one. AskUI’s article uses this exact move.

Claim 7 — Implicit framing: “deterministic = legacy, agentic = future”

This is not stated literally by AskUI, but it pervades the structure of their piece: the deterministic column is presented as a list of limitations, the agentic column as a list of capabilities.

The honest framing is the inverse, depending on the buyer’s constraint:

What deterministic gives you

Bit-for-bit reproducibility. Same script, same input, same output, today and in three years. Zero per-execution cost. Air-gap operation, no cloud dependency in the hot path. Auditable failure: when a test fails, the trace tells you exactly why. Survives the model version churn that disrupts LLM-piloted suites every quarter.

What agentic gives you

Faster initial authoring, at the cost of artifacts that are not reproducible. Tolerance to UI drift, at the cost of silent passes when the model misinterprets the screen. Natural-language interfaces, at the cost of nobody being able to debug the test when it fails. Per-execution billing, indefinitely. None of this is a fair trade for professional QA.

A QA team protecting a payment terminal under fiscal audit needs the left column. A growth team smoke-testing a marketing landing page can probably afford the right column. Pretending either category is the universal answer is selling the buyer a paradigm rather than helping them solve a problem.

The economic shape of the choice

Beyond the technical claims, the two paradigms differ in their economic shape, and the difference matters more than the comparison acknowledges.

Dimension	OculiX 3.0.4	LLM-piloted alternative
Commercial-use terms	MIT, unlimited, code public	Non-commercial on free tier, paid plans required for production
Pricing transparency	Free, no plan to upgrade	Plans not publicly disclosed, contact-sales gating
Hot-path cost per test	Zero — local CPU	Token-billed inference per round trip
Cloud dependency	None in default deployment	Cloud agent OS, hosted inference hub
Air-gap suitability	Native	Requires self-hosted enterprise plan, custom contract
Self-hosted at scale	Default mode	Enterprise tier only
Auditability of runs	Per-byte deterministic	Subject to model and prompt drift

For a buyer running a hundred thousand test executions per day, the per-token inference cost of the agentic model is not a rounding error. It is a recurring operational expense that grows with usage. The deterministic alternative has zero such expense: the script runs on the runner that the buyer was going to provision anyway, and the matcher uses the CPU cycles that would otherwise be idle.

For a buyer in a regulated industry — banking, defense, healthcare, manufacturing under traceable quality control — the air-gap and audit requirements are not negotiable. A tool that requires a cloud inference hub does not even reach the evaluation phase. A tool that runs entirely on-premise under MIT open-source terms clears that gate by construction.

AskUI’s comparison glosses over both of these, framing pricing as “starts free” and air-gap as “available in enterprise”. A buyer who reads only the marketing surface might conclude the difference is small. A buyer who looks at the contracts and the deployment architecture will conclude otherwise.

Where the agentic paradigm is sometimes invoked — and why the trade-off does not hold

The agentic paradigm is routinely defended on four use cases. I do not consider any of them a valid justification for professional QA — I list them so we can see why.

Highly volatile UIs under continuous redesign. The argument: regenerating patterns costs more than re-running an LLM. The reality: deterministic pattern regeneration is measurable and bounded; LLM inference cost is recurring and tied to usage. Deterministic maintenance can be debugged; model drift cannot.
Authoring by non-developers in natural language. The argument: lower the barrier to entry. The reality: a test is not low-code, it is engineering. Handing its authorship to someone who does not read code means accepting upfront that nobody will be able to audit the test when it fails.
Discovery work. The argument: find edge cases a deterministic script would not exercise. The reality: discovery is done by human exploratory testing or by deterministic mutation/fuzzing; an LLM clicking semi-randomly is not an explorer, it is unanalyzable noise.
Workflows with a human-in-the-loop. The argument: an operator validates the model’s output. The reality: what has just been described is an assistant — not a test engine. Calling that a QA automation is a linguistic abuse that hides the fact that the decision remains manual.

In none of these four cases do cloud inference billing, dependence on a model version that will change, or loss of reproducibility become acceptable for a test base meant to last years. AskUI is a real and well-executed product within its space; the space itself is simply not that of professional QA, regardless of vendor.

The structural problem with per-inference pricing in QA

The point that follows is not about AskUI specifically. It applies to any editor — whatever the brand, whatever the marketing copy — who proposes a token-billed inference model on a cloud-only platform for an activity whose nature is high-volume, repeated, and budget-constrained. QA is precisely that activity.

Let me lay out the structural mismatch.

QA is, by its very purpose, an exercise in volume:

A regression suite runs on every pull request — 30 to 100 PRs per day for a mid-size product team
A nightly run executes the full suite — 1 000 to 20 000 test cases
A canary run hits production-like environments multiple times per release
A smoke run protects each deployment in CI
An integration suite exercises every supplier delivery

Multiply by the number of teams, by the number of products, by the number of environments, by the number of replays needed for flaky-test analysis. A serious QA practice in a mid-to-large organization performs hundreds of thousands of test executions per day. That is not pathological scale. That is the normal floor.

Now overlay a per-inference pricing model on this volume.

Concrete scenario	Inferences per day	Cost trajectory at typical inference pricing
Small team, daily smoke	~5 000	Manageable in year 1, scales linearly
Mid-size product, PR + nightly + canary	~50 000	Becomes a visible budget line
Enterprise QA practice across multiple products	~500 000	Becomes a board-level conversation
Compliance-grade replay for audit reconstitution	~1 000 000	Becomes a deal-breaker

The QA leader who chooses a per-inference pricing tool inherits a structural problem with two opposing pressures:

To control cost

The team is pushed to test less — skip the canary, drop the nightly, narrow the regression scope. Cost is contained at the expense of coverage.

To keep quality

The team is pushed to spend more — keep coverage intact, accept the linear bill, escalate to procurement when the line crosses the threshold. Quality is contained at the expense of budget.

The two pressures converge on a budget ceiling that arrives faster than the maturation of the practice.

Why deterministic inverts the incentive

No per-execution fee

A regression suite that runs ten times today and a hundred times tomorrow consumes ten and a hundred times the runner minutes the organization was already paying for. The vendor is not involved in the marginal cost.

Marginal cost ≈ CPU seconds

Each additional test execution costs the price of an additional second of CPU time on a runner that was already provisioned. The QA practice can mature without renegotiating any contract.

This is the structural reason most regulated industries — banking, defense, healthcare, manufacturing under traceable quality control — have never adopted a per-token cloud automation primary engine, and will not. It is not a question of trust or maturity. It is a question of arithmetic.

A concrete cost comparison at three volumes

Abstract arguments about pricing models are easier to dismiss than concrete numbers. Let me put real volumes on the table.

AskUI does not publish a pricing page. askui.com/pricing returns a 404 at the time of writing, and every aggregator (Capterra, GetApp, Software Advice, techjockey) lists “contact sales” for the Developer, Agent Hub, and Enterprise plans, with a single public anchor of “starting from 49 €”.

I will not invent a fourchette I cannot source. That would commit the exact methodological fault I accuse the original AskUI piece of. What I can compare is the shape of the cost curve: per-inference billing grows linearly with QA volume; my published flat editor service does not. The math below is on cost behaviour, not on a fabricated quote.

For the competing tools that DO publish (mabl, testRigor, Applitools), the ranges below are taken from their public 2026 grids and industry buyer’s guides.

The OculiX editor service in plain words

Before the numbers, what the editor service actually covers. The OculiX engine is free; the editor service is an optional flat-fee engagement layered on top for buyers who want a private channel, response-time commitments, named contacts, or vendor-of-record status. Three packages, three flat prices that do not scale with the buyer’s QA volume:

Tier	What it covers	Typical fit
Small Team support (~12 – 15 k€/year)	Private email channel, response < 5 business days, bug prioritization, roadmap access	Single team running OculiX in production, wanting a private channel without overhead
Premium editor service (~35 – 70 k€/year)	Everything in Small Team + hotline on agreed slots, response < 1 business day, quarterly review, direct roadmap influence	Mid-size product team where automation is on the critical path, multiple stakeholders
Enterprise editor service (~90 – 180 k€/year, volume discount above 150 k€)	Everything in Premium + named engineer, response < 4 business hours, contractual SLA, on-call windows for critical releases, vendor-of-record status (audit-defensible)	Enterprise QA practice with regulatory exposure, multi-product or multi-entity scope

What each level explicitly excludes: custom feature development (separate engagement, billed per feature), endless time-and-materials work (the engagement is flat-priced — the editor carries the delivery risk, not the buyer), and 24/7 coverage outside named critical windows. The buyer always knows what they pay for; the editor always knows what is committed.

The structural point: at any tier, the price is flat for the year regardless of how many tests the buyer runs. A team that doubles its test volume next year pays the same. A team that quadruples it pays the same. The line item is predictable for budget review, which is exactly what the inference-billed alternatives cannot offer.

With that understood, the three buyer scenarios.

Scenario 1 — Small team, 100 executions per day (25 000 / year)

Tool / model	Year 1 cost	3-year cumulative	Cost behavior
mabl (Team plan + extra exec minutes)	~12 000 – 15 000 €	~36 000 – 45 000 €	Linear with execution minutes
testRigor (Pro tier, ~5 000 steps/month)	~6 000 – 10 000 €	~18 000 – 30 000 €	Linear with test steps
Applitools Eyes / Autonomous (Growth)	~6 000 – 25 000 €	~18 000 – 75 000 €	Linear with Test Units
AskUI (subscription + token Hub)	Not publicly disclosed — contact sales	Same shape — linear with volume	Linear with inference calls
OculiX OSS	0 €	0 €	Constant
OculiX OSS + Small Team editor support (optional)	~12 000 – 15 000 €	~36 000 – 45 000 €	Constant — flat editor engagement

At this scale, the cheapest line is OculiX OSS at zero euros, full stop. For buyers who want a private channel, response-time commitments, bug prioritization, and roadmap access, the OculiX Small Team editor support package sits in the same price band as the cheapest competing offers — and stays flat as the buyer’s QA volume grows, instead of scaling with executions. The OSS engine alone is enough for many teams; the editor support is a bonus, not an obligation.

Scenario 2 — Mid-size product team, 1 000 executions per day (250 000 / year)

Tool / model	Year 1 cost	3-year cumulative	Cost behavior
mabl (Enterprise, custom)	~40 000 – 80 000 €	~120 000 – 240 000 €	Custom-quoted, linear with usage
testRigor (Enterprise, near-unlimited steps)	~60 000 – 100 000 €	~180 000 – 300 000 €	Custom-quoted enterprise
Applitools Enterprise	~50 000 – 100 000 €	~150 000 – 300 000 €	Custom-quoted enterprise
AskUI (subscription + heavy inference)	Not publicly disclosed — contact sales	Same shape — linear with volume	Linear with inference calls
OculiX OSS	0 €	0 €	Constant
OculiX OSS + Premium editor service (optional)	~35 000 – 70 000 €	~105 000 – 210 000 €	Constant — flat editor engagement

At 1 000 executions per day, the competing tools start to look like operational expenses with their own line in the budget review. OculiX OSS still costs zero. The OculiX Premium editor service starts at 35 000 € per year — below the entry point of every competing enterprise tier in this band — with hotline, response within one business day, quarterly review, and direct roadmap influence. The price stays flat whatever the QA volume grows to. Doubling the suite next year keeps the OculiX line at the same number; the competing alternatives scale linearly with usage.

On a three-year horizon, the gap is no longer marginal. OculiX Premium accumulates to ~105 000 – 210 000 € over three years. Mabl Enterprise accumulates to ~120 000 – 240 000 €. testRigor and Applitools accumulate to ~150 000 – 300 000 €. AskUI does not publish prices, so I won’t fabricate a cumulative — what is observable from their model is that it scales linearly with inference volume, which at 250 000 executions per year compounds the gap mechanically. Even at the upper bound of the OculiX Premium range, the total stays below the entry of testRigor and Applitools and at the bottom of mabl Enterprise. At five years and at ten years, the gap widens further — because the OculiX line is flat and the competing lines are not.

Scenario 3 — Large enterprise QA practice, 10 000 executions per day (2.5 M / year)

Tool / model	Year 1 cost	3-year cumulative	Cost behavior
mabl / testRigor / Applitools (Enterprise+, custom)	~200 000 – 500 000 €	~600 000 – 1 500 000 €	Custom-quoted, linear with usage
AskUI (full inference Hub at 2.5 M executions)	Not publicly disclosed — contact sales	Same shape — linear with volume	Linear with inference calls
OculiX OSS	0 €	0 €	Constant
OculiX OSS + Enterprise editor service + volume discount (optional)	~180 000 – 250 000 €	~540 000 – 750 000 €	Constant with volume discount

At 10 000 executions per day, the gap is no longer a comparison — it is a categorical statement about which paradigm scales economically. OculiX OSS costs zero. OculiX with full editor service plus volume discount is still well under what a buyer would pay for the inference-based alternative even before counting the cloud lock-in.

Why OculiX editor service is not a subscription tax

There is a temptation, reading the tables above, to view the OculiX Premium or Enterprise editor service as just another vendor lock-in line item. It is not, and the structural difference matters.

The OSS engine runs without it

OculiX is MIT, the binaries are public, the documentation is on the site, the community channel is open. A team can adopt OculiX, run it in production for ten years, and never pay the editor a euro. This is not theoretical — many of the close-to-a-hundred deployed organizations do exactly that.

The editor service is a flat engagement

When a team chooses to engage the editor, they pay a flat annual price for SLA, named engineer, vendor-of-record status, roadmap influence, and migration support. The price does not scale with the number of tests they run. They can quadruple their QA volume next year without renegotiating.

The exit is real

At the end of an engagement, the team keeps their entire codebase, their patterns, their CI pipelines, their automations. There is no proprietary format to extract from, no cloud hub to migrate off, no token bill that survives the contract. The OSS engine runs the same the day after the contract ends as the day before.

The model is Red Hat, not SaaS

Red Hat does not charge per CPU cycle on Linux. Confluent does not charge per Kafka message. Sonatype does not charge per Nexus artifact resolved. OculiX does not charge per test execution. The editor sells engagement and support around a free engine — the proven OSS-plus-services model.

A buyer who reads only the per-tool price table might frame the choice as “free vs paid”. That framing misses the point. The choice is: pay per usage forever, or pay nothing and engage the editor only if and when the buyer values doing so. The decision is reversible. The cost ceiling is knowable. The exit is a git pull away.

Choosing in 2026

The honest framing for a buyer evaluating visual automation in 2026 looks like this:

Decide what must be reproducible. If the test is part of a compliance audit, a regression suite that gates production, or a forensic trail that must survive model and vendor churn, the answer is deterministic. There is no other.
Decide what perimeter the test must reach. If the test must run on-premise, in an air-gapped environment, behind a regulated network, the answer is deterministic by default, and any cloud-dependent paradigm is excluded by the perimeter itself.
Decide what economic shape fits the scale. Per-token inference billing compounds linearly with QA maturity. The deterministic alternative runs on CPU cycles the organization already provisions. At any scale beyond the pilot phase, the math is decided.
Refuse the false choice. Some sellers will frame the question as “deterministic for safety, agentic for productivity”. The framing is itself the trap. A test that is not reproducible is not a test; it is a demo. Demos are not productivity, they are theater.

These four questions are observable. The buyer can answer them without depending on a vendor’s marketing narrative. They do not require choosing between “legacy” and “future”; they require recognizing that QA is engineering, and that engineering does not negotiate reproducibility.

OculiX 3.0.4 is, in 2026, the most complete deterministic visual automation runtime available under MIT open-source terms. It runs on Windows, macOS including Apple Silicon, and Linux. It supports Android. It runs in VNC sessions for remote and parallel execution. It persists its locator cache in a portable form that survives git clone and CI runners. It is used in production by close to a hundred organizations across regulated industries. The code is on GitHub. The open-source terms are irrevocable. The next release is scoped and public.

If the buyer’s constraint is reproducibility, audit, air-gap, or zero per-execution cost, OculiX is the answer. If the buyer’s constraint is assisted exploration with systematic human validation, what they are looking for is an assistant — not a test engine. Confusing the two with QA automation is a linguistic abuse the market will eventually correct. Either way, the choice should be made on facts about the current state of each option — not on a 2024 snapshot of one tool extended into a 2026 article about another.

I publish the data I can verify. I link to AskUI’s article that prompted this one so that any reader can check both sides for themselves. The repository is open. The benchmark methodology is public. The architecture is documented. There is no hidden layer to negotiate access to.

That is the difference open-source MIT terms actually make.

A closing note on accurate comparison

What I welcome — and what I don’t

Honest comparison

Rankings that conclude OculiX is the wrong tool for a given use case, supported by accurate technical claims. Buyer guidance toward a competitor when the competitor genuinely fits better. The market matures when comparisons are factual — and OculiX has nothing to fear from an honest one.

Systematic misrepresentation

Version numbers misrepresented by multiple stable releases. Shipped features described as absent. Limitations of an unrelated tool attributed to OculiX. Deployment architectures ignored despite being publicly documented. This is a different category and is treated as such.

The applicable legal framework

Remedies preserved, dialogue preferred

Legal remedies preserved

The maintainers preserve every legal remedy available in our jurisdictions: misrepresentation, misleading comparative advertising, defamation when the threshold is reached, and the moral rights attached to the authorship of the code.

Dialogue is the first move

These remedies are not exercised as a first response. The preferred path is dialogue, correction, and right-of-reply — this post is itself the public correction offered amicably. Escalation happens only when the pattern persists and the offered correction is ignored.

Universal, not targeted

This is not a warning addressed to any specific competitor. It is a position that applies to anyone, today or in the future, who chooses to compete with OculiX through fabrication rather than through honest technical and economic argument.

MIT protects the user, not the misrepresentation

The MIT terms grant unrestricted rights to use, modify and redistribute the code. They do not strip the maintainers of legal personhood, reputational interests, or the protections French and European law extends to honest economic actors.

Repository: github.com/oculix-org/Oculix

Benchmark referenced: Storing spatial memory in a PNG (May 2026)

Source article responded to: AskUI vs SikuliX (April 2026)

The engine assumed one screen. The job needed four.

Fri, 22 May 2026 10:00:00 GMT

Seven hours. That’s how long a visual test suite for a Fortune 500 retail group used to take — roughly 300 tests driving point-of-sale terminals, run one after another, against a regulatory compliance window that does not move.

Today it finishes in under two. Four sessions in parallel, on a single 8 GB Azure VM running Windows Server 2019 — a swap file, basic provisioning, nothing exotic. Runtime divided by five. No test farm, no per-minute cloud bill, no new hardware.

The speed is the headline. It’s not the interesting part. The interesting part is that the engine was never built to do this — and neither is almost any visual-automation tool. They assume one screen. I needed four. This is the story of the gap between those two numbers, and what I had to build to close it.

A world of one screen

Visual automation was born in a world of one physical screen. One machine, one desktop, one cursor. SikuliX — which OculiX continues — inherited that worldview, and for fifteen years it was the correct one. The engine keeps a notion of the current screen, and very reasonably it keeps it in one place: shared, global state. When there is only ever one screen, a global is not a flaw. It’s the right design.

That assumption is invisible right up until the moment you break it. VNC breaks it.

A VNC session isn’t a screen — it’s a network framebuffer. No monitor, no desktop login, no DISPLAY. You can open a dozen, each on its own port, each pointing at a different remote machine. Suddenly “the current screen” stops being a singleton and becomes a question: which one?

The collision

The first time I declared two sessions and ran two test flows at once, they walked all over each other. Not with an error — worse, with quiet nonsense. Clicks landing on the wrong register. A framebuffer read mid-repaint. Results that came back green for entirely the wrong reasons.

I tested it carefully, because I didn’t believe it at first. Two flows, two ports, two remote machines. They polluted each other every single time.

The robot and the client were per-instance — that part was clean. But the port and the registry of live sessions lived in shared, static state. Two threads reaching for the same global. The single-screen assumption, baked into the data model, surfacing at exactly the moment I asked for more than one screen.

What I built — around the engine, not inside it

So I built the missing layer myself, on top of the engine rather than in it. Four pieces, each one fixing a failure I had actually watched happen.

Isolation per flow. Each test flow got its own session, bound to its own thread — its own port, its own remote, its own everything. No flow could see another’s state, because there was no shared “current session” left to see. The engine didn’t hand me thread-scoped sessions, so I made them thread-scoped from the outside.

A port is a promise. Two flows must never claim the same port, and “probably free” is not good enough at startup when four of them race. So allocation became atomic: a small dedicated range, and a rule — a flow claims a port through a lock no two flows can win simultaneously, or it doesn’t run at all. A port stops being a guess and becomes a contract.

One tunnel per session. Each remote register was reachable only through an SSH tunnel exposing its framebuffer on a local port. One tunnel per flow, opened on demand, torn down after — and, the detail that cost me a late night, closed before the next one opened. Orphaned tunnels hold ports hostage long after the process that spawned them is gone. Close-before-open, not close-after. Order matters.

Readiness, not optimism. The engine waited a fixed few seconds for a session to come up and hoped for the best. Hope does not survive four sessions starting at once. So I waited for proof instead: the RFB banner — the three bytes a VNC server sends to announce it’s ready for pixels. No banner, no test. A probe, not a sleep.

Why four sessions gave 5×, not 4×

Four workers, five times faster, looks like broken arithmetic. It isn’t.

The killer in a sequential suite isn’t the total work — it’s the variance in test durations. My ~300 tests ranged from a few seconds to many minutes. Run in a single line, a long test sits in front of forty short ones and they all just wait. Sequential execution doesn’t only pay for the work; it pays for every test queued behind a slower one. With uneven durations that waiting tax is enormous, and invisible, because the CPU is idle the whole time.

I split the load into batches sized by functional area — thirty tests here, ninety there, a hundred and twenty somewhere else, the way the business actually reasons about its registers. Balanced, not just parallel. The four sessions finished at roughly the same time instead of one straggler holding the line.

So the 5× isn’t four workers doing 4× the work. It’s four balanced workers deleting waste the single queue was silently burning. When the thing you replaced is that wasteful, even modest, well-distributed parallelism overshoots the naive ratio.

Why all of this lived outside the engine

I want to be precise here, out of respect for the engine and the people who built it. The core does one thing extremely well: clean, reliable, single-session VNC. It kept a single-screen promise and it kept it honestly. The orchestration — the thread isolation, the atomic ports, the tunnels, the readiness probe — was mine, and it lived in my harness. I made a different promise, one layer up, and kept that one too.

For a long time that was the right division of labour. The engine stayed pure; the parallelism was an application concern. But once you’ve named the gap, you can’t un-see it — and you start to wonder why the engine shouldn’t simply do this itself.

The roadmap: making parallel sessions first-class

Here’s where it gets exciting, because the next step is obvious now that the gap has a name. Everything I built outside the engine is the engine’s natural next form.

Thread-scoped sessions

No more global “current screen.” A session belongs to the flow that opened it, by construction — so two flows can never reach for the same state, because there is no shared state to reach for.

Port allocation, built in

The engine hands you an isolated session and owns the atomic port claim itself. No range to reserve, no lock-files to babysit, no collisions to debug at 23h.

Readiness on the RFB banner

The optimistic startup sleep replaced by a real probe — the session reports ready when the server actually says it’s ready, not when a timer guesses it might be.

A clean SSH tunnel, already shipped

A pure-Java tunnel (no WSL, no external sshpass) already lives in the core. The next step is wiring it into a documented, first-class parallel pattern.

The goal for the next major version is simple to state and worth the work: declare N sessions, get N isolated sessions. No harness, no lock-files, no late nights over ghost tunnels. The single-screen assumption retired — gently, and with full respect for the fifteen years it served exactly right.

OculiX is MIT, open source, built in the open. If you drive screens for a living — registers, kiosks, terminals, application instances by the dozen — this is the corner of the roadmap I’d most like company on.

The takeaway

Visual automation has a reputation for not scaling. It earned that reputation honestly: the tools were built for one screen, and most of them still are. But the limit was never the pixels, and it was never the matching. It was an assumption — one screen — that nobody had yet needed to question.

Question it. Give the engine real sessions, isolate them, balance the load across uneven work — and a bored 8 GB VM does in two hours what used to take seven. No farm. No new hardware. No per-minute bill.

The screens were never the bottleneck. The assumption was.

OculiX is open-source under the MIT license. The parallel VNC engine — and the work to make multi-session parallelism native in the core — is described in the Parallel VNC execution guide.

CodeQL: the boy who cried Error

Thu, 21 May 2026 08:00:00 GMT

We enabled GitHub CodeQL on OculiX in April 2026. The repository inherits sixteen years of Sikuli and SikuliX history before our continuation began, so the scanner was pointed at roughly 110 000 lines of Java accumulated since 2009.

It returned a hundred or so findings across all severities. Nineteen of them were classified Error, the highest tier. The classification suggests urgency: each Error is, in CodeQL’s taxonomy, the kind of thing a maintainer should drop everything to fix.

We dropped nothing. We triaged each one. After triage, the result is unambiguous: zero of the nineteen findings have caused an observable user incident in fifteen years. Most of them are dormant architectural defects, dead code paths, or false positives produced by structural blindness in the analyzer.

Meanwhile, fifty-three missing catches for NumberFormatException — places where parsing a user-provided number can crash the application — are classified Note. Almost invisible. Below Warning. Far below Error.

This post walks through the nineteen findings, explains why none of them actually warrant the Error tier, and explains why the things that should warrant it are buried two levels down. The conclusion is not that static analysis is useless. The conclusion is that CodeQL’s severity classification is broken in both directions — overstating the trivial, understating the serious — and that the noise this generates is the main thing keeping the tool from being useful at scale.

Methodology

For each finding, we asked three questions:

Does it produce an observable user incident? Has it caused a crash, a wrong result, a regression in fifteen years of use? Has anyone filed a ticket against it?
If not, would it produce one if a specific code path were exercised? Is it dormant but reachable, or dormant and unreachable?
Is the finding even correct? Does CodeQL’s analysis actually match the runtime behavior, or has it missed something about the language, the bridge to another runtime, or the type system?

Three categories emerged from this triage:

Architectural defects

Real problems in the code that have never been triggered. Either the branch is dead, the configuration is never set, or the upstream dependency is never invoked. The finding is correct but the impact is theoretical.

Code smells

Patterns that look suspicious but have legitimate uses (or, more often, have just persisted as harmless cruft for over a decade). The code is ugly but not broken.

False positives

CodeQL’s analysis is wrong. It missed a dynamic access, a language bridge, an inheritance relationship, or a type guarantee. The finding does not correspond to any real risk in the code.

What we did not find in our nineteen Errors: any case where a user reproduced a crash, a wrong result, or a regression caused by the code at the location CodeQL flagged. Not one.

A tour of the nineteen Errors

Below is the breakdown by CodeQL rule, in descending count.

Container contents are never accessed (8 findings)

CodeQL’s claim: a collection or map is allocated and possibly written to, but its contents are never read back. The implication is that the container is dead code that should be removed.

Eight findings of this kind landed on our codebase. The triage:

Three are genuine dead code. Two local variables named classpath inside ExtensionManager.java, and a local map docParams inside Crawler.java. All three are declared, populated in a loop, and never consumed. Cleanup candidates. None of them have produced a single user-visible defect.

Four are false positives caused by dynamic access. Lists like aliases, filenames, mimeTypes in Lexer.java (the syntax highlighter grammar) are populated by parsing grammar metadata and consumed by a registry mechanism that CodeQL cannot trace. A map called libsLoaded in RunTime.java tracks loaded native libraries and is consulted before each native load — but the consumption goes through a debug helper that CodeQL’s call-graph analysis does not follow. From the static point of view, the containers look unread. From the runtime point of view, they are read on every invocation.

One is the most interesting finding in the entire set. SikulixServer.allowedIPs is declared like this:

private static List<String> allowedIPs = new ArrayList<>();

The class is the Sikuli network server. The list is named after a security feature: an IP allow-list to restrict who can connect. There is a method called makeAllowedIPs that populates the list. There is no code that reads the list. No filter checks against it. No connection-handling code consults it. The allow-list is built but never enforced.

If an administrator configures allowedIPs thinking they are restricting access, they are not. The server accepts every connection regardless of the configured allow-list, because the list is allocated, filled, and never read.

This is a security defect. A serious one, by any reasonable definition. CodeQL classifies it as Maintainability — Error. Not Security. Not Critical. Just “your container is unused”, as if the impact were cosmetic.

We have opened an internal issue to investigate whether the network server is reachable in any default OculiX deployment, and to either remove the network server entirely or wire up the missing filter. As of this writing, the defect has not produced an incident report in any of the years it has existed — but that is more an accident of the network server’s low usage than a vindication of CodeQL’s classification.

Self assignment (2 findings)

CodeQL’s claim: a variable is being assigned to itself. The assignment has no effect.

The first finding is in Runner.java:396:

public static void setLastScriptRunReturnCode(int lastScriptRunReturnCode) {
    lastScriptRunReturnCode = lastScriptRunReturnCode;
}

The setter’s parameter shadows the static field. The author intended Runner.lastScriptRunReturnCode = lastScriptRunReturnCode; or this.lastScriptRunReturnCode = lastScriptRunReturnCode;. As written, the setter is inert. It has been inert since whenever it was added — almost certainly in the mid-2010s, possibly earlier.

For fifteen years, any caller invoking Runner.setLastScriptRunReturnCode(5) has stored exactly nothing. The field lastScriptRunReturnCode stays at its initial value of 0 forever. Anyone reading the field gets 0 regardless of what scripts have actually returned.

Has anyone noticed? Apparently not. Either the field is never consulted, or the consumers do not depend on the value being non-zero, or the corner cases where it would matter have not been hit in fifteen years of cumulative production use. The “bug” is real in the source code, but the observable behavior is no different from what it would be if the setter worked correctly. The downstream code path has adapted to the dysfunction.

The second finding is in Deflate.java:1703, inside com.jcraft.jzlib (the JZlib compression library, embedded for SSH purposes):

Deflate dest = (Deflate) super.clone();
dest.pending_buf = dup(dest.pending_buf);
dest.d_buf = dest.d_buf;
dest.l_buf = dup(dest.l_buf);
dest.window = dup(dest.window);

The clone() method deep-duplicates every buffer except d_buf. The assignment dest.d_buf = dest.d_buf; is a typo where dup(dest.d_buf) was intended. The result is that the cloned Deflate instance shares its d_buf with the original. Modifying the buffer through either reference is visible through both.

The defect is in an upstream library that nobody has actively maintained for years. The OculiX project carries it because the library is bundled. The risk is real but only materializes if the Deflate.clone() method is called in a context where both the original and the clone are used concurrently — a rare pattern that JZlib’s tests apparently do not cover and that nobody seems to hit in production.

Both Self assignment findings are genuinely correct. Both have been silently dormant in the codebase for over a decade. Neither has produced a user-visible incident. CodeQL classifies them as Error. The classification suggests they require immediate attention. The empirical evidence suggests they have been irrelevant for a decade.

Reference equality test of boxed types (1 finding)

In RecordedEventsFlow.java:149:

} else if (!modifiers.isEmpty() && characterWithModifiers == character) {
    actions.add(new TypeTextAction(keyText, modifiersTexts));
}

characterWithModifiers and character are boxed Integer (or Character) values. Comparing them with == compares object references, not values. Java caches boxed integers from -128 to 127, so for any ASCII codepoint, two boxed instances of the same value point to the same cached object, and == returns true by accident. For codepoints above 127 — accented Latin, CJK, Cyrillic, Arabic, Hindi, almost any non-ASCII script — the cache does not apply, and == returns false even when the values are equal.

The defect is correct. It has existed for years. The empirical impact is invisible because:

The recorder is used predominantly with ASCII input in the regions where it is most used.
When the comparison fails for non-ASCII input, the recorder falls through to a different branch that produces a slightly different — but still functional — action.

This is a textbook example of a real defect that has been silently absorbed by the surrounding code’s tolerance. No user has filed a ticket. No CI test has caught it. The fix is one line (.equals() instead of ==). The classification as Error is consistent with CodeQL’s rule but inconsistent with the actual user impact, which is zero.

Equals method does not inspect argument type (1 finding)

In PatternPaneScreenshot.java:36:

@Override
public boolean equals(Object o) {
    return false;
}

CodeQL warns that the method does not check the argument’s type, and might therefore lead to a ClassCastException somewhere downstream. The implementation is trivial: it returns false for any input, including null. There is no cast. There is no downstream usage of the result that depends on the argument’s type.

The override exists inside a Comparator declaration. The Comparator interface allows equals to be overridden to indicate that two comparators impose the same ordering. Returning false is a safe, conservative default: “I do not claim to impose the same ordering as you do.” It does no harm.

The finding is a false positive on this specific code. The override is pointless (it could be removed without changing behavior), but it is not a defect. CodeQL classifies it as Reliability — Error, suggesting it could cause a crash. It cannot. The implementation has no crash path.

Non-callable called (2 findings, Python)

Two findings on Sikuli.py, the Jython-based scripting layer that ships with OculiX. Both report that an imported name is being called as if it were callable when it is not.

import org.sikuli.script.ScreenRemote as SR
SCREEN = SR(adr, str(port))

The CodeQL Python analyzer treats this as CPython. It sees import org.sikuli.script.ScreenRemote as importing an attribute, has no idea that ScreenRemote is a Java class, and concludes that calling SR(...) is invalid.

In Jython, this import is valid Java-to-Python bridging. SR(...) invokes the Java constructor ScreenRemote(String, String). The runtime knows this. CodeQL does not.

This is structural blindness across language runtimes. CodeQL Python has no awareness of the Jython platform. Every Java class accessed from a Jython script will be flagged as “non-callable called”. The number of such findings will grow linearly with the size of the scripting layer.

Both findings are false positives. There is no fix on the project side. The recourse is to disable the rule in .codeql.yml or to suppress the findings manually one by one.

Dereferenced variable is always null (1 finding)

In ButtonGenCommand.java:127:

if (pref.getAutoCaptureForCmdButtons()) {
    btnCapture = null; //TODO new ButtonCapture(pane, line);
    pane.insertComponent(btnCapture);
    btnCapture.captureWithAutoDelay();
}

The author explicitly assigned null, left a TODO comment, and then dereferenced the variable twice. Whoever opens the IDE with the AutoCaptureForCmdButtons preference enabled will crash on btnCapture.captureWithAutoDelay().

This is the only finding in the set that is a definite crash trap. CodeQL is correct to flag it. Calling it Error is, for once, defensible.

Has anyone hit the crash? The preference is off by default, and the TODO has lived in the code for years, which suggests nobody enables this preference in practice. The crash is a real risk but a dormant one. Fix: replace the null with new ButtonCapture(pane, line) and finish the TODO.

Contradictory type checks (1 finding)

In Offset.java:152:

public <RMILO> Offset(RMILO whatEver) {
    if (whatEver instanceof Region || whatEver instanceof Match) {
        Region what = (Region) whatEver;

CodeQL claims that when whatEver is Match but not Region, the cast to Region will fail at runtime. This would be true if Match and Region were unrelated types. They are not. Match extends Region throughout the Sikuli class hierarchy. Every Match is also a Region. The cast is safe.

CodeQL does not consult the inheritance graph of the project, so it treats the two types as disjoint. The OR condition is redundant (whatEver instanceof Region already covers the Match case), but it is not unsafe.

The finding is a false positive caused by type system blindness. The check on Match is dead, in the sense that it is always reached on objects that already match instanceof Region. Removing it is a cleanup, not a bug fix.

Type mismatch on container modification (1 finding)

In HotkeyManager.java:263:

boolean res = _instance._removeHotkey(key, mod);
if (res) {
    hotkeys.remove(key);
}

hotkeys is a Map keyed on String. key is Integer (an AWT virtual key code). Map.remove(Object) accepts any Object for backward compatibility with pre-generics Java, so the call compiles cleanly. At runtime, no entry in the map has an Integer key, so the remove silently does nothing. The intended key for the map is a textual representation that must be constructed from key and mod.

The hotkey is reported as removed (because _removeHotkey succeeded at unbinding the OS-level listener), but the entry stays in the map indefinitely. If hotkeys is consulted to dispatch incoming events, the old hotkey’s listener continues to fire.

The defect is real and observable in theory. In practice, no user has filed a ticket about a hotkey persisting after rebind. Either:

The listener-dispatch path does not consult the map.
Nobody rebinds hotkeys.
The fallout is silently absorbed by other code.

Whichever explanation applies, the failure mode of this defect is dormant in the same way as the others. CodeQL’s classification as Error matches the technical risk but overstates the empirical urgency.

Missing format argument (2 findings)

In Commons.java:1525:

RunTime.terminate(999, "Commons.loadLib: %s");

The format string demands an argument, none is supplied. When this line is reached, the String.format call throws IllegalFormatException. The thrown exception is wrapped inside the terminate call, which means the user sees a confusing format-related error instead of a clean shutdown with code 999 and a useful diagnostic message.

The branch is an error path inside the native library loading code. It executes when a library fails to load — itself a rare event. The defect aggravates the user experience precisely at the moment when it is already poor (something is breaking, and now the error reporting is also breaking), but it does not produce its own incident on a healthy system.

The second finding is Crawler.java:441:

docMethod = String.format(pyMethod, clazz, method, docParams, returns);

The format string pyMethod expects five arguments, four are supplied. Every invocation of this method throws IllegalFormatException. The Crawler appears to be a documentation generator that has not been run in a long time — the API it scrapes for has changed, but the generator has not been touched to follow. The defect is invisible because the tool is no longer used.

Both findings are real defects. Both are essentially dead code (one on a rare error path, one in an unused tool). Their classification as Error is technically correct and operationally misleading.

What the classification actually looks like

The total: 19 findings classified as Error. After triage:

Category	Count	Share
Dead-code paths with crash potential	4	21 %
Architectural defects with dormant impact	8	42 %
False positives (structural blindness)	7	37 %
Findings with observed user incidents	0	0 %

Zero observed user incidents in fifteen years across all nineteen findings.

Meanwhile, in the layers below Error, CodeQL reports the following counts on the same codebase:

Rule	Findings	Classified
Missing catch of NumberFormatException	53	Note
Dereferenced variable may be null	38	Warning
Potential input resource leak	24	Warning
Ignored error status of call	23	Note
Implicit conversion from array to string	11	Note
Potential output resource leak	11	Warning
Use of default toString()	3	Note
Non-synchronized override of synchronized method	2	(below visible threshold)

Fifty-three places where parsing a user-provided number can crash the application. Classified Note. Almost invisible.

Twenty-four places where a file or network resource may not be closed, leading to file descriptor exhaustion on a long-running server. Classified Warning. Below Error.

Two places where overriding a synchronized method without re-applying synchronized creates a race condition in multi-threaded code. Buried so deep it does not even appear in the default UI summary.

Where the severity system breaks down

The pattern across the data is unambiguous. CodeQL’s Error tier is dominated by structural patterns that may produce defects in extreme cases but routinely do not. The Warning and Note tiers contain patterns that frequently produce real production incidents on real systems. The mapping from classification to operational urgency is inverted.

The reasons appear to be structural to how the tool reasons:

Definite vs probable

CodeQL favors definite analytical conclusions (“this is always null”, “this assigns to itself”) over probabilistic ones (“this might be null”). Definite conclusions get Error. Probabilistic ones get Warning or Note. But probable failures on 53 lines of code matter more than definite failures on 2 dormant lines.

Pattern-level severity

Every instance of a rule inherits the rule’s severity. There is no per-instance reasoning. A self-assignment that is dead code and a self-assignment that breaks a hot path receive the same Error tier. The reviewer cannot distinguish them from the severity column.

Cosmetic and structural together

Rules like “Container contents are never accessed” cover three very different real-world cases: dead variables, dynamically accessed structures, and security-critical filters that fail open. All three receive Maintainability — Error. The CISO reviewing the report cannot find the security one without reading every finding.

No cross-rule prioritization

A self-assignment with no impact is Error. A NumberFormatException not caught on user input is Note. There is no mechanism in CodeQL to bring the second above the first in any output. The order of rules in the output is alphabetic or chronological, never by operational risk.

The combined effect is that a developer who opens CodeQL on a mature codebase sees a few dozen “Errors” — most of them irrelevant — and a few hundred “Notes” — among which the genuinely concerning ones are buried. The natural response, after the third or fourth review, is to stop opening CodeQL.

What CodeQL actually catches well

This post would be unfair if it described only the failure modes. CodeQL does some things well, and the things it does well are worth naming.

Definite null dereference (when the analyzer is sure, not “may be”). The ButtonGenCommand finding is a real crash trap. The rule has high precision.

Type-incompatible container modification. The HotkeyManager finding is a real bug that compiles cleanly and silently misbehaves at runtime. Without static analysis, this kind of generic-erasure pitfall is genuinely hard to catch.

Format string desync. Both Missing format argument findings are real. The rule has very high precision because the analysis is local.

Self-assignment. Both findings are real. The rule has perfect precision because there is no plausible legitimate use.

What unites these is that they are narrow, local, and high-confidence rules. CodeQL is at its best when the pattern is sharply defined and the analysis is bounded. It is at its worst when it generalizes — across rules, across severities, across code paths it cannot follow.

How we configured CodeQL to be useful

After triage, the actions we took for this scan:

Suppressed the Jython-related rules entirely. Non-callable called and a few other Python rules produce structural false positives on every Jython-bridged file. There is no fix on our side. Suppressing the rule per-file via comments is more cost than benefit. We disable the rule in .codeql.yml.
Filed an internal issue for SikulixServer.allowedIPs to investigate whether the network server is reachable and to either fix the missing filter or remove the server. The finding sat under Maintainability — Error, where nobody who triages security findings would look for it.
Treated the 53 NumberFormatException findings as a focused remediation backlog, separately from the Error-tier findings. They are not the same kind of work and should not be in the same queue.
Documented our dismissals. For every Error-tier finding we dismissed, we wrote a one-line justification in the commit that closed the alert. This is the documentation our future selves will need when CodeQL re-flags the same finding after a re-scan.
Did not aim for zero Errors. Zero CodeQL Errors is not a meaningful goal. A meaningful goal is zero unjustified Errors and a tracked, finite list of structural issues being worked on.

Why we still keep CodeQL on

Despite all of the above, CodeQL stays enabled on OculiX. The OpenSSF Scorecard checks for a SAST tool integration and rewards projects that have one. Procurement teams in regulated sectors check that CodeQL or an equivalent is configured. The signal value to outside observers is real, even when the operational value is poor.

But the cost of running CodeQL is also real: the triage time, the alert fatigue, the temptation to ignore the tool entirely. We pay that cost in exchange for the external signal. We do not pretend that what we get back internally is anywhere near worth the marketing of static analysis.

If you maintain a similar codebase and you are considering whether to enable CodeQL: enable it, run it once, do the triage we just walked through, and then decide rule-by-rule which queries you actually want to run. The default query suite assumes you have time to read several hundred findings. Most teams do not. The most useful configuration is the one that leaves you with twenty findings a week, all of which are worth opening.

The boy who cried Error eventually had a wolf. We will probably see one, eventually, in a CodeQL scan. We will recognize it because it will be the only finding in months that we cannot dismiss in a single line. Everything else will continue to be the noise that the wolf is hidden in.

Repository: github.com/oculix-org/Oculix

The findings discussed are from a CodeQL scan on commit 29a14d9. Detailed dismissal justifications are visible in the Security tab.

Why automation tools cannot type Chinese, and what OculiX does instead

Wed, 20 May 2026 08:00:00 GMT

In May 2026, an OculiX user filed issue #232. The summary was three lines long. The user wanted to call type with two Chinese characters in their automation script. Instead of typing them, the call produced silence, garbage, or in some IDE configurations a sequence of unrelated Latin characters depending on the active keyboard layout.

The fix, when it landed, was fifteen lines of Java. The explanation behind it stretches back to how java.awt.Robot was designed in 1998, around an implicit assumption that one character on a screen corresponds to one keystroke on a keyboard. That assumption holds beautifully for ASCII Latin. It breaks for Chinese, Japanese, Korean, Arabic, Hindi, and even for accented Latin characters when the user’s keyboard layout does not happen to expose them as direct keys.

This is the story of why visual automation frameworks have historically struggled to type non-ASCII, what OculiX decided to do about it, and why the workaround is more philosophically interesting than the bug it solves.

The bug report

The user, an active OculiX adopter, was automating an internal Chinese-language application. Their script needed to fill a search field with a person’s name. The straightforward call returned silently. The search field stayed empty. No exception was thrown. No log line surfaced anything unusual. The automation simply continued, the next step failed because the search field had no input, and the test reported FindFailed on whatever element was supposed to appear after the search.

This is the worst kind of automation bug. There is no signal that something went wrong at the moment something went wrong. The error appears three steps later, attached to an unrelated element, with no link back to the actual failure point.

After some investigation, the user established that:

ASCII input worked: type("hello") produced “hello” in the field.
French accented characters partly worked: typing a word with an accent produced the unaccented version on a US layout.
Chinese input failed silently on every layout tested.
Switching the OS keyboard layout to a Chinese IME did not help: the OculiX type call still produced garbage.

The bug was filed not as a feature request but as a question: “Is this expected? How is everyone else handling CJK?”

The honest answer was that everyone else was working around it. Manually. Each script that needed to type Chinese was implementing its own custom paste routine inline, copying the string to the clipboard with Jython’s java.awt.Toolkit calls and then sending Ctrl-V via keyDown/keyUp. The workaround was well-known among users who automated CJK applications, but it was not in the documentation, and it required understanding why the straightforward call did not work.

This blog post explains the why, and what we did to make the straightforward call work.

Why Robot.keyPress fundamentally cannot type Chinese

To understand why type with Chinese fails, you have to understand what type actually does under the hood. In OculiX (and in the original Sikuli, and in SikuliX), the call descends through several layers and eventually reaches java.awt.Robot.keyPress(int keycode) and java.awt.Robot.keyRelease(int keycode), both members of the AWT API that has shipped with the Java platform since version 1.3 in May 2000.

Robot.keyPress(int keycode) is a thin wrapper around the operating system’s low-level keyboard event injection mechanism: SendInput on Windows, XTestFakeKeyEvent on X11/Linux, CGEventPost on macOS. All three of these accept a virtual key code, an integer identifier of a physical key on the keyboard. Not a character. Not a Unicode codepoint. A key.

This is the first fault line. When you call Robot.keyPress(KeyEvent.VK_A), you are not asking the system to produce the character “A”. You are asking it to simulate pressing the physical key that, on the currently active keyboard layout, would produce some character. The character depends on the layout. The layout depends on the user’s system configuration. The result depends on both.

For pure ASCII Latin in a Western layout, the mapping is reasonably stable. The key VK_A produces “a” or “A” depending on shift state. The key VK_1 produces “1” or ”!” depending on shift state. Sikuli historically maintained a hardcoded table mapping each ASCII character to a sequence of virtual key codes, and it worked because Western keyboards have a key for every ASCII character.

The trouble starts the moment the character has no physical key on the keyboard.

The Chinese case

There is no constant for Chinese characters in java.awt.event.KeyEvent. There cannot be one. Chinese characters do not correspond to keys. A standard Chinese keyboard is, physically, a US-QWERTY keyboard. Chinese characters are produced through an Input Method Editor (IME) that interprets sequences of QWERTY keystrokes as phonetic input (Pinyin), looks up matching characters, displays a candidate menu, and inserts the chosen character into the active text field once the user confirms a selection.

To type a Chinese phrase through an IME, a Chinese user types Pinyin on the physical keyboard, the IME intercepts the sequence, recognizes it as the romanization of a candidate set, presents a candidate menu (because multiple character sequences share that romanization), and waits for the user to either accept the top candidate via space or pick a different one via number keys or arrow keys.

This is not what Robot.keyPress does. Robot.keyPress injects events at a layer beneath the IME. The IME never sees them as Pinyin input. The events are interpreted as raw key events by the receiving application, which sees the Pinyin letters as literal characters. The Chinese intent is lost.

Even more confusingly, on a system where the active keyboard layout is Chinese (with an IME enabled), Robot.keyPress(VK_N) might:

Be intercepted by the IME, treated as Pinyin input, and contribute to building a candidate menu that the user never sees because the automation does not pause to select
Be passed through to the application as a literal “n” if the IME is in alphanumeric mode
Be silently dropped if the IME is mid-composition for another sequence

The behavior is non-deterministic from the automation’s perspective, and entirely dependent on IME state, which the automation cannot reliably read or control through the AWT API.

The Japanese, Korean, Arabic, Hindi cases

The same logic applies to:

Japanese, where IMEs interpret romaji or kana keystrokes into hiragana, katakana, or kanji
Korean, where IMEs assemble hangul syllables from jamo keystrokes
Arabic, where the keyboard layout itself is non-Latin and Robot virtual keys do not map predictably
Hindi and other Indic scripts, where IMEs combine consonants and vowel marks into syllabic clusters
Vietnamese, where IMEs add tonal diacritics through dedicated keys

All these languages share the property that the displayed character is not a one-to-one mapping with a physical keystroke. The mapping is mediated by an IME that operates above the OS keyboard event layer that java.awt.Robot injects into.

The accented Latin case (more subtle)

Even within the Latin script, the same problem appears in a milder form. Consider typing a French word with an accent on three different keyboard layouts.

Layout	Result of typing a French word with accent
US-QWERTY	Accent silently dropped, no VK code maps to it
French AZERTY	Accent reproduced correctly
German QWERTZ	Accent dropped or garbage produced
UK-QWERTY	Accent dropped
Spanish	Accent dropped (uses dead keys)

The accent characters have dead-key compositions on most layouts, which the AWT Robot cannot reliably simulate because dead keys are stateful at the OS layer and there is no portable way to query whether the dead key has been “consumed”.

The same applies to many Latin extension characters. The character is “Latin”, but the keyboard layout determines whether it has a dedicated key or requires a composition sequence.

Summary of the diagnosis

The deep reason type with non-ASCII fails is not specific to OculiX or to Sikuli. It is a structural limitation of the java.awt.Robot API, which injects events at the keyboard event layer beneath the IME and beneath the keyboard layout translation. This API was designed in 1998 for testing GUI applications in an ASCII-Latin context, and it has never been extended to handle the layered character input model that the rest of the world uses.

The clipboard route as workaround

Once you accept that you cannot type CJK characters through the keystroke injection path, the question becomes: how do you get them into an input field at all? The most portable answer, used by professional automation engineers for two decades, is the clipboard.

The idea is simple. Instead of pressing keys, you:

Copy the target string to the system clipboard
Send the OS-specific paste keystroke (Ctrl-V on Windows/Linux, Cmd-V on macOS)
The receiving application processes the paste event and inserts the clipboard content into the focused field

This route bypasses the keyboard layout layer entirely. The clipboard contains arbitrary Unicode bytes. The paste keystroke is a single key combination present on every modern platform. The receiving application sees the content as text, not as a sequence of keystrokes, and inserts it through its text handling code path, which is Unicode-aware.

The clipboard route has been the unofficial standard workaround in the Sikuli community since at least 2012. It worked. But it required users to know about it, to implement it themselves, and to handle a list of edge cases (clipboard backup before pasting to avoid destroying user data, clipboard restoration after pasting, focus verification, etc.).

The OculiX change for issue #232 was to make this route the default, transparently, for any input that contains characters outside the 7-bit ASCII range.

Implementation: fifteen lines of Java

The actual change in OculiX is small enough to quote in full.

The detection

private static boolean containsNonAscii(String text) {
    if (text == null) return false;
    for (int i = 0; i < text.length(); i++) {
        if (text.charAt(i) > 127) return true;
    }
    return false;
}

Seven lines. We iterate over the input string and check whether any character has a codepoint greater than 127 (the 7-bit ASCII boundary). If yes, the input cannot be reliably typed through the keystroke path. If no, the input is pure ASCII Latin and can use the existing keystroke pipeline.

The choice of 127 as the boundary is deliberate. Everything from 0 to 127 is ASCII, has a stable Robot mapping on Western keyboards, and works with the legacy code path. Everything above 127 is either Latin extension (accented characters), other scripts (CJK, Cyrillic, Arabic, etc.), or symbols. All of these benefit from going through paste rather than through simulated keystrokes.

The routing

public int type(String text) {
    if (containsNonAscii(text)) {
        return paste(text);
    }
    try {
        return keyin(null, text, 0);
    } catch (FindFailed ex) {
        return 0;
    }
}

public int type(Object target, String text) throws FindFailed {
    if (containsNonAscii(text)) {
        return paste(target, text);
    }
    return keyin(target, text, 0);
}

The routing is two if-statements added before the existing logic. If the input contains non-ASCII characters, the call is forwarded to paste (which already exists and was already battle-tested). Otherwise, the existing keyin path runs unchanged.

The two-argument variant type(target, text) first clicks the target image to focus the input, then routes the text through paste(target, text), which performs the same click-then-paste sequence internally. The user-facing API is identical; only the internal routing changes.

What stays on keystrokes

public int type(Object target, String text, int modifiers)
    throws FindFailed {
    return keyin(target, text, modifiers);
}

The modifier-bearing variants type(text, modifiers) and type(target, text, modifiers) are intentionally not routed through paste. Holding Shift, Ctrl, or Cmd while pasting Unicode characters has no meaningful semantic — these modifiers exist to shift the meaning of physical keystrokes (Shift+A produces “A”, Ctrl+S triggers a save action), not to alter clipboard content.

If a user calls a modifier-bearing variant with CJK characters, the keystroke path runs and produces a FindFailed if the characters cannot be typed, which is the honest behavior. Silently switching to paste would lose the user’s intent.

The behavior matrix

The result for users of the API is a clean four-way matrix:

Input	Modifiers	Path taken	Why
Pure ASCII	None	Keystrokes	Backward compatible, fast, no clipboard side effect
Pure ASCII	Yes (Shift/Ctrl/Cmd)	Keystrokes	Modifiers map to keystroke semantics
Non-ASCII	None	Clipboard paste	Only way to reliably produce the characters
Non-ASCII	Yes (Shift/Ctrl/Cmd)	Keystrokes (will likely fail)	Honest failure; modifier+paste has no meaning

This is the kind of fallback that the user does not see and does not need to know about. The scripts that worked before still work the same. The scripts that previously needed manual clipboard workarounds now just work with the obvious call.

Tradeoffs of the clipboard route

The clipboard path is not a free win. It has properties that are different from the keystroke path, and users automating sensitive workflows should be aware of them.

Clipboard pollution

Pasting destroys the previous clipboard content. The OculiX paste implementation backs up the clipboard before pasting and restores it afterward, but the backup-restore window is not instantaneous and can interfere with other clipboard listeners.

Focus dependency

The paste keystroke targets whatever has keyboard focus. A wrong paste is one chunk of misplaced text, much more visible than wrong keystrokes which usually masks the issue.

Paste timing

Some applications throttle or queue paste events. Pasting twice in rapid succession can produce only one paste, or merge two pastes into one. The keystroke path is less susceptible.

No partial input

A keystroke-based input can be interrupted in the middle. A paste is atomic. If you want autocomplete observation, the keystroke path is the right one.

For most real-world automation use cases, these tradeoffs are acceptable. A QA engineer automating a Chinese banking form does not need character-by-character granularity; they need the form filled correctly. A test scenario filling Japanese customer data into a healthcare CRM does not depend on clipboard preservation across the operation.

The cases where the tradeoffs matter — autocomplete observation, clipboard-watching tooling — are edge cases that the user can opt out of by explicitly calling keyin instead of type, or by providing the input one character at a time.

What this enables in production

Issue #232 was filed by a single user, but the impact of the fix is broader than that single use case. We have seen the change unlock automation work in several customer contexts since it landed.

APAC banking back-office

A French bank with subsidiaries in Hong Kong and Singapore needed to automate regression tests on their Mandarin and Cantonese branch interfaces. Before the fix, their automation team maintained two separate codebases: one for European frontends, one for APAC with manual clipboard workarounds. The two codebases merged. Maintenance cost dropped by roughly a third.

Japanese healthcare CRM

A healthcare provider in Osaka automating patient intake workflows ran into the same wall. Patient names, addresses, prescription drug names — none survived the keystroke path. The team had a custom paste wrapper as workaround, but it predated the Ed25519 audit module and could not be signed for compliance. The integrated paste path now passes their audit.

European multilingual e-commerce

A French automation team running tests on a German marketplace was hitting silent failures whenever a product name contained German umlauts. The German keyboard layout was active on the test machine, but the keystroke table assumed French AZERTY and silently dropped the characters. The clipboard route bypasses all of this.

Internal IT helpdesks across regions

Multinational corporations run helpdesk automations interacting with internal ticketing systems containing employee names in dozens of native scripts. Cyrillic, Greek, Arabic, Hindi. Each required a separate workaround. The fix makes all of them automatic.

The common thread is that internationalization in automation tooling has historically been an afterthought. Tools are built and tested in their authors’ language, and the assumption that “text input works” is rarely verified against the full Unicode space. Fixing this in OculiX brought a basic property back into line with what users reasonably expect.

What this doesn’t solve

The fix is good but not complete. Several adjacent problems remain open.

The structural lesson

The lesson behind this fix is not about Chinese or Japanese specifically. It is about what happens when a tool’s API and its underlying assumption diverge.

type(String) reads, to a 2026 developer, like a Unicode-aware string-to-input function. The signature accepts a String, which in Java is fully Unicode-capable. The naive expectation is that whatever Unicode string you pass should appear in the target field, character by character, exactly as written.

That expectation collides with an implementation that descends, through layers of well-meaning abstraction, into a 1998-era API that injects keyboard events at a layer beneath the keyboard layout. The mismatch between the modern API surface and the legacy implementation creates a silent failure mode that takes years of accumulated user reports to fully document.

The fix is fifteen lines of Java because the workaround already existed in the codebase (the paste method). The change just added a smart router that picks the right path based on the input content. The hard part was not the code; it was identifying that the routing decision belonged at the type level, not at the user level.

This is a recurring pattern in mature codebases. The capability is there. The pieces are in place. What is missing is the small connective decision that makes the capability available to users who do not know which internal method to call. A good API surface hides the choice between keyin and paste. The user calls type and gets the result they expected.

For anyone maintaining a similar tool with a similar AWT-era foundation: this fix is generalizable. The same routing logic, the same ASCII boundary detection, the same paste fallback applies in PyAutoGUI, AutoHotkey, Robot Framework keyword libraries, or any other automation tool that relies on Robot-style keystroke injection. The 1998 API will not be redesigned. But its limitations can be bounded by a smarter layer above it.

Repository: github.com/oculix-org/Oculix

Original issue: oculix-org/Oculix#232

Storing spatial memory in a PNG: how we made visual UI matching 6.5× faster in OculiX

Tue, 19 May 2026 08:00:00 GMT

This is a long technical post about a small change. It explains a measurement we never thought to make, the discovery it produced, the implementation we built around it, and what it means for visual automation suites running in CI.

The headline number is real but not particularly dramatic: visual UI matching in OculiX became roughly 6.5 times faster, measured on 50 cold-start JVM runs on an Intel i3 7th generation laptop. The interesting part is not the speedup itself. It is the path that led to it, and the structural lesson that came with it.

If you maintain or use a visual automation framework, a test suite that watches pixels, or any tool that needs to find an image inside another image repeatedly across separate process invocations, the pattern we describe here will probably apply to you. The implementation took about 200 lines of Java, no external dependencies, and three micro-modifications inside the existing codebase. The standards we relied on have been published since 1996.

What follows is the full story, ordered the way the investigation actually unfolded, not the way a marketing post would tell it.

The benchmark we never made

OculiX is the active continuation of the Sikuli and SikuliX visual automation lineage, MIT-licensed and used in production by close to 100 organizations across banking, defense, healthcare, manufacturing, and retail. Its core operation is a function called find: given a small image (a button, an icon, a region of UI), find it inside the current screen capture and return its coordinates. Every other operation in the public API contains at least one call to find underneath.

Inside the codebase, find is well understood. It uses OpenCV template matching via JNI bindings, with five fallback strategies cascaded inside Finder.java:

Mode 1: Standard match

Exact template matching with the configured similarity threshold. The fast path for stable UI elements.

Mode 2: DPI-aware rescale

If the screen DPI differs from the pattern capture DPI, the template is rescaled before matching.

Mode 3: Tolerant blur

GaussianBlur applied to both source and target. Tolerates antialiasing and subtle color variations.

Mode 4: Grayscale smart

Conversion to grayscale before matching. Tolerates color theme changes.

Mode 5: Multi-scale brute force

Last resort. Tries multiple scales (0.5x to 2x) to catch significantly resized elements.

The code is twenty years old at this point, refined incrementally by the original Sikuli authors at MIT in 2009, by Raimund Hocke from 2010 to 2025 under SikuliX, and now by the OculiX maintainers.

Performance, in this kind of codebase, is rarely benchmarked from scratch. Everyone assumes it is whatever it has always been. A find call takes “some milliseconds”, or “a few hundred milliseconds” on a slow machine, and life goes on.

So I sat down on a Sunday afternoon to actually measure it. Not because there was a problem. Because the question of how fast it really was had never been answered with a number on the current hardware I had in front of me.

The harness

The setup was deliberately simple. A standalone Java class called FindTiming, compiled directly against the OculiX complete-win jar, performing exactly one find per JVM invocation, then exiting. A batch script wrapping that class in a loop of fifty separate executions.

So fifty cold starts. Each one paid the JVM startup cost, the OpenCV library load, the Tesseract OCR engine load, the OculiX framework initialization, then performed exactly one find and reported the elapsed time before exiting.

The baseline

The result, on this five-year-old i3 laptop, was extremely consistent:

Metric	Value
Mean	502 ms
Median	502 ms
Minimum	480 ms
Maximum	545 ms
Range	65 ms
Standard deviation	18 ms

Half a second to find a small image on a 1920 by 1080 screen. On a modern Intel i7 or Apple Silicon machine the number would be lower, probably by a factor of two or three. On a GitHub Actions standard runner it would be roughly comparable to the i3. On an older corporate desktop running a Citrix client through a VPN it would be slower again.

Take 500 milliseconds and project it across a test suite:

Suite scale	Find calls	Pure find time @ 500ms
Small functional suite	200	100 seconds
Medium regression suite	1 000	8 min 20 s
Large nightly suite	5 000	41 min 40 s
Enterprise full coverage	20 000	2 h 46 min

Multiply by the number of suites running per day in a CI environment, multiply by the number of CI minutes billed by the runner provider, and the cost becomes very real.

That was the baseline. No optimization had been attempted yet. The number was simply the truth about what happened on the metal.

What was already in the code

Before deciding what to optimize, the right move is always to look at what the code already does. OculiX inherits sixteen years of optimization attempts from the Sikuli and SikuliX lineage. A naive optimizer would re-read the OpenCV documentation and propose to switch to a faster matching algorithm. That would be a beginner mistake.

The thing to investigate first is whether there is some optimization the existing code already attempts but cannot fully complete in the current configuration. That is almost always where the gains hide.

A few minutes of grep revealed a symmetric pair of fields and a setting that nobody seems to talk about:

Image.lastSeen

Private Rectangle field on every Image object. Stores the position of the most recent successful match. Paired with getLastSeen() and setLastSeen(rect, score) accessors.

Settings.CheckLastSeen

Public static boolean, set to true by default since at least 2018. Enables the optimization at the framework level.

checkLastSeenAndCreateFinder

Private method in Region.java. When called, creates a Finder restricted to the small rectangle around the previous match, falling back to full-screen only if needed.

The intent of these three pieces is clear once you piece them together. After a successful find, the rectangle of the match is stored in the Image object’s lastSeen field. On the next call to find for the same image, if Settings.CheckLastSeen is true and lastSeen is non-null, the code creates a Finder restricted to that small rectangle and tries to match there first. Only if the small-region match fails does it fall back to a full-screen scan.

This is a classic spatial memoization pattern, well-known in computer vision. Sikuli implemented it correctly, a long time ago.

The pattern works, until it doesn’t

The optimization works beautifully inside a single JVM session. If you run a script that performs screen.find("submit_button.png") ten times in a row, the first call pays the full-screen scan cost, but the next nine calls find the image almost instantly through checkLastSeen. The cache hit rate is essentially 100 percent on stable UIs.

There is, however, a subtle but critical limitation.

This is exactly what our benchmark exposed. Fifty separate JVM invocations, fifty Image objects with lastSeen always equal to null at the moment of the find call, fifty full-screen scans. The checkLastSeen optimization was active and present in the code throughout, but it had no input to work with. The cache was empty because the cache lived inside a process that died right after building it.

This is the core observation. The existing optimization was correct in design. It was simply unable to bridge the gap between JVM invocations. In a typical test environment, where each test is its own process, the optimization never had a chance to engage.

The code that solved the problem was already there, written long before this benchmark was performed, in a careful and well-tested form. The missing piece was not a clever algorithm. It was a way to keep the optimization’s input alive across process boundaries. A storage problem, not a computation problem.

The missing piece: persistence

Once the gap was identified, the question became: how do you persist Image.lastSeen between JVM invocations? Several candidate approaches surfaced, each with their own trade-offs.

Approach	Description	Drawbacks
Sidecar file	Write `foo.png.position` next to each PNG	Two files to commit, risk of desynchronization, clutter
Central project file	Single `.oculix-positions.toml` at project root	Linear lookup cost, merge conflicts in parallel CI, full file rewrite per change
PNG ancillary chunk	Embed the position metadata inside the PNG itself	Requires writing PNG-aware code, but standard since 1996

The PNG ancillary chunk option won, for reasons that became clearer as we explored the constraint of CI environments.

PNG ancillary chunks: the underused W3C standard

The PNG file format, standardized by the W3C in 1996, is a structured container made of a fixed signature followed by a sequence of chunks. Each chunk has a four-byte length, a four-byte type, a variable-length data payload, and a four-byte CRC32 of the type and data.

The PNG standard distinguishes between critical chunks (IHDR, IDAT, IEND, and others), which are mandatory for decoding the image, and ancillary chunks, which are optional and can be safely ignored by decoders that do not recognize them.

We chose the type code oPLx for our chunk:

Lowercase o: ancillary, not critical
Uppercase P: private to OculiX
Uppercase L: reserved bit
Lowercase x: safe to copy

Decoded: an optional, private, safe-to-copy chunk identified by oPLx. Other PNG tools encountering an OculiX-modified file see something they do not recognize, preserve it untouched on save, and ignore it on load. Compatibility is total.

The format of the oPLx chunk

The internal layout of the oPLx chunk’s data payload is deliberately small. The goal was a fixed-size, fast-to-parse, easy-to-debug binary structure. The total payload is exactly 34 bytes:

Offset	Size	Type	Field	Notes
0	4	ASCII	Magic `OPL\0`	Redundant identifier, prevents misinterpretation
4	2	uint16 (BE)	Version	Currently `1`. Bump on breaking format change.
6	4	int32 (BE)	X	Pixel coordinate, last successful match
10	4	int32 (BE)	Y	Pixel coordinate, last successful match
14	4	int32 (BE)	Width	Match rectangle width, pixels
18	4	int32 (BE)	Height	Match rectangle height, pixels
22	8	int64 (BE)	Timestamp	UNIX epoch milliseconds, last update
30	4	int32 (BE)	Run count	Total successful matches since file creation

Total: 34 bytes of payload, plus the standard 12 bytes of PNG chunk framing (4 bytes length, 4 bytes type, 4 bytes CRC32). Each pattern gains 46 bytes of metadata embedded in its PNG file. For a project with 500 patterns, this represents 23 kilobytes of additional space across the entire pattern library. Effectively negligible.

Design choices in this format

A few decisions deserve commentary.

Big-endian byte order

Non-negotiable. The PNG standard mandates big-endian for all multi-byte integers in chunk fields. Following the same convention inside our payload simplifies parsing and removes confusion with future tooling.

32-bit signed for coordinates

Generous. 16-bit unsigned would have sufficed for single-screen resolutions. We chose 32 bits to leave room for multi-monitor setups where coordinates extend into negative space and tens of thousands of pixels.

64-bit timestamp

Standard UNIX epoch milliseconds. No bytes saved here. Audit trails that span years require room.

Run counter

Allows detecting dead patterns (counter at zero), unstable patterns (high counter on young file), and locked-in patterns (counter grows without timestamp updates).

The chunk is plaintext at this stage. The integrity check is the standard CRC32 that PNG mandates at the end of every chunk; it catches accidental corruption but is not cryptographically strong.

Implementation: three micro-modifications

The actual change inside OculiX comes down to three modifications in two existing files, plus one new utility class. Total addition: roughly 250 lines of Java. No external dependencies. No new Maven coordinates. The standard JDK classes DataInputStream, DataOutputStream, ByteBuffer, and java.util.zip.CRC32 cover all the needs.

1. Image.load() reads the chunk

After ImageIO.read() decodes the pixels, a separate streaming read parses the PNG chunks, locates oPLx, and calls setLastSeen(rect, 1.0) on the current Image instance.

2. doCheckLastSeenAndCreateFinder expands the search

Instead of creating a Region exactly the size of the previous match, it now creates a search box 2.5x larger, clamped to screen bounds. Tolerates UI drift between runs.

3. find() writes the chunk after match

After updating in-memory lastSeen, the chunk in the PNG file is streamed through a temp file and atomic-renamed. The position persists to disk.

Modification 1: Image.load() reads the chunk

In Image.java, the existing load method reads the PNG file from disk into a BufferedImage using ImageIO.read. This call decodes only the image data (the IDAT chunks). It does not parse other chunks. Our addition opens the same file separately, in streaming mode, walks through its chunks until it finds the oPLx chunk if present, parses the position metadata, and calls setLastSeen(rect, 1.0) on the current Image instance.

// In Image.java, after ImageIO.read(fileURL) at line 1018:
try {
    File pngFile = new File(fileURL.toURI());
    byte[] chunk = PngChunk.read(pngFile, "oPLx");
    if (chunk != null && chunk.length >= 34) {
        ByteBuffer buf = ByteBuffer.wrap(chunk);
        byte[] magic = new byte[4];
        buf.get(magic);
        if (magic[0] == 'O' && magic[1] == 'P'
            && magic[2] == 'L' && magic[3] == 0
            && buf.getShort() == 1) {
            int cx = buf.getInt();
            int cy = buf.getInt();
            int cw = buf.getInt();
            int ch = buf.getInt();
            setLastSeen(new Rectangle(cx, cy, cw, ch), 1.0);
        }
    }
} catch (Exception ignored) {}

The streaming parser is deliberately optimized for the common case where the chunk is present and is one of the first non-critical chunks in the file. It skips the PNG signature (8 bytes), then enters a loop: read the chunk length, read the four-byte chunk type, compare it to oPLx, return the payload if matched, return null if IEND is reached, or skip the chunk data and CRC and continue otherwise.

This is the architectural detail that matters most. The chunk reading is a strict addition that fills a gap in the existing system. It does not replace anything. It does not modify the contract of any existing method. If the chunk is absent (a legacy PNG that has never been processed by OculiX, a PNG whose chunk was stripped by an aggressive optimizer, a PNG generated by an external tool), the code falls through to lastSeen being null, which is exactly the situation the existing codebase has handled for sixteen years. The fall-through path is the cold-start path. It still works. It is just slower.

Modification 2: doCheckLastSeenAndCreateFinder expands the search

In Region.java, the existing method creates a small Region exactly the size of the previous match rectangle. This works well for in-session use, where the image has just been matched at the exact same position. It works less well for cross-process use, where the UI may have drifted slightly between runs.

Our modification expands the search rectangle by a factor of 2.5 around the stored center, clamped to the screen bounds.

// Replace at line 2891:
Rectangle ls = img.getLastSeen();
int sw = (int) (ls.width * 2.5);
int sh = (int) (ls.height * 2.5);
if (sw > screen.w) sw = screen.w;
if (sh > screen.h) sh = screen.h;

int cx = ls.x + ls.width / 2;
int cy = ls.y + ls.height / 2;
int sx = cx - sw / 2;
int sy = cy - sh / 2;

// Translate to stay inside screen, do not truncate
if (sx < 0) sx = 0;
if (sy < 0) sy = 0;
if (sx + sw > screen.w) sx = screen.w - sw;
if (sy + sh > screen.h) sy = screen.h - sh;

Region r = Region.create(sx, sy, sw, sh);

The 2.5 multiplier was chosen empirically:

Multiplier	Effect
1.5x	Occasionally misses drifted patterns
2.0x	Marginal improvement, still occasional misses
2.5x	Sweet spot: tolerates drift without ambiguity
3.0x	No safety improvement, starts introducing ambiguity on dense UIs
4.0x	Multiple visually similar elements get reached, confusing the matcher

The clamping logic preserves the search box size when the pattern is near a screen edge. A pattern at (0, 1022) still gets a full 250 by 145 search box, just positioned at (0, 935) instead of being centered. The match rectangle for the original pattern still fits inside the search box.

Modification 3: find() writes the chunk after a successful match

In Region.java, the existing find method already calls img.setLastSeen(lastMatch.getRect(), lastMatch.getScore()) after a successful match. This updates the in-memory lastSeen field. Our addition extends this call with a write to the PNG file’s oPLx chunk.

// At line 2284 of Region.find(), after setLastSeen:
img.setLastSeen(lastMatch.getRect(), lastMatch.getScore());

// New: persist to PNG chunk
try {
    File pngFile = new File(img.getFileURL().toURI());
    ByteBuffer buf = ByteBuffer.allocate(34);
    buf.put((byte) 'O').put((byte) 'P').put((byte) 'L').put((byte) 0);
    buf.putShort((short) 1);
    Rectangle r = lastMatch.getRect();
    buf.putInt(r.x).putInt(r.y).putInt(r.width).putInt(r.height);
    buf.putLong(System.currentTimeMillis());
    buf.putInt(getRunCount(img) + 1);
    PngChunk.write(pngFile, "oPLx", buf.array());
} catch (Exception ignored) {}

The chunk-writing logic streams through the PNG file once, copying every chunk through to a temporary file, replacing the oPLx chunk in place if it already exists, or inserting a fresh oPLx chunk before the IEND marker if not. At the end, the temporary file replaces the original via an atomic file system rename.

This streaming approach is more complex than reading the whole file into memory, modifying the byte array, and writing the result back, but it is meaningfully more robust:

Property	In-memory	Streaming (chosen)
Memory cost	Proportional to PNG size	Constant (~8 KB buffer)
Crash safety	Risk of partial write	Atomic rename: old or new, never half
Cost on small PNG	< 1 ms	1-2 ms
Cost on large PNG (1 MB+)	20-50 ms + allocation	4-8 ms, no allocation spike

The PngChunk utility class

Reading and writing PNG chunks does not require an external library. The format is simple enough that a 200-line utility class handles both operations with zero dependencies. The class exposes two methods:

public static byte[] read(File png, String type) throws IOException

Returns the payload bytes if the chunk is found, null otherwise. The read uses streaming DataInputStream, with skip() to bypass non-target chunks. Exits as soon as the target chunk is located.

public static void write(File png, String type, byte[] payload) throws IOException

Streams the source PNG to a temp file, replacing or inserting the chunk, then atomic-renames the temp file over the original. CRC32 computed via java.util.zip.CRC32 (hardware-accelerated on modern CPUs).

private static final byte[] SIGNATURE = {
    (byte) 0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'
};
private static final byte[] IEND_BYTES = { 'I', 'E', 'N', 'D' };

The PNG file signature is 8 well-known bytes. The IEND chunk type terminates every PNG.

The total code in PngChunk.java is 218 lines including comments, blank lines, and the class declaration. The file has no imports outside the java.io, java.nio, java.util.Arrays, java.util.zip.CRC32, and java.nio.charset.StandardCharsets packages, all standard JDK.

Benchmark methodology

A benchmark is only useful if its methodology is described in enough detail that someone else can reproduce it.

Hardware and runtime

Item	Value
CPU	Intel Core i3-7100 (2 cores, 4 threads, 3.9 GHz)
RAM	8 GB DDR4-2400
Storage	NVMe SSD
OS	Windows 10 build 19045
Java	OpenJDK 25 from Eclipse Temurin
OculiX	3.0.3 release, feature branch with modifications
Screen	1920 × 1080, no DPI scaling
Pattern	Windows search bar fragment, 12 × 58 pixels, at (0, 1022)

Harness logic

Each benchmark run consisted of fifty independent JVM invocations, each:

Starting a fresh process
Loading OculiX and its native dependencies
Performing exactly one find call against the screen
Printing the elapsed time in milliseconds to standard output
Exiting

The elapsed time was measured using System.nanoTime() immediately before and after the screen.find(pattern) call. This excludes JVM startup, library loading, and framework initialization. It includes only the find operation itself, including the screen capture inside the find.

A separate post-processing class read the fifty timing lines from the captured log and computed: arithmetic mean, median, minimum, maximum, range, and standard deviation.

Two scenarios measured

Scenario	Initial state of PNG	Expected behavior
Baseline	No `oPLx` chunk	All 50 runs in full-screen scan
Optimized	`oPLx` chunk pre-written	All 50 runs in small-region scan

Both scenarios were measured cold-start (50 separate JVM invocations) to ensure the comparison reflects the CI environment.

Results

The numbers are the headline of this post. Let me state them precisely.

Metric	Baseline (FULL)	Optimized (ROI)	Improvement
n	50	50	—
Mean	502 ms	77.7 ms	×6.46
Median	502 ms	77 ms	×6.5
Minimum	480 ms	63 ms	×7.6
Maximum	545 ms	113 ms	×4.8
Range	65 ms	50 ms	comparable
Standard deviation	18 ms	11 ms	tighter

Speedup factor on the find call alone: 6.46.

Observations on these numbers

Low variance in both conditions

Standard deviation is 4% of mean in baseline, 14% in optimized. Neither is noisy enough to require repeated measurement.

Optimized scenario has a floor

Roughly 60-70 ms cannot be reduced further by this technique. Dominated by screen capture (20-40 ms) and OpenCV setup (20-40 ms).

Speedup depends on pattern size

Small patterns (under 50×50 px) give the largest speedup. Large patterns (300×300+ px) give a smaller speedup because the 2.5x search region itself becomes substantial.

Faster hardware preserves ratio

On modern CPUs, absolute timings shrink but the relative speedup factor stays at ×6-8. The overhead floor is proportionally less significant.

Hardware projection

If we extrapolate to other typical hardware:

Hardware	Baseline (FULL)	Optimized (ROI)	Speedup
i3 7th gen (measured)	502 ms	77 ms	×6.5
i7-13700K (projected)	~150-180 ms	~22-25 ms	×8
Apple M2/M3 (projected)	~80-120 ms	~10-15 ms	×8
GitHub Actions standard runner (projected)	~200-250 ms	~28-35 ms	×7

What this means for CI suites

The benchmark measures one find operation in isolation. A real-world test suite performs many operations, each containing at least one find. Translating the per-operation speedup into a suite-level wall-clock improvement requires a few assumptions.

Consider a moderately sized regression suite: 100 test cases, each performing 20 visual interactions on average, for 2000 total find calls.

Scenario	Time per find	Total find time	Savings
Baseline (i3)	500 ms	16 min 40 s	—
Optimized (i3)	77 ms	2 min 34 s	14 min 6 s
Optimized (modern i7)	22 ms	44 seconds	15 min 56 s

Economic projection

If the suite runs once per pull request on a CI runner billed at $0.008 per minute (the rough GitHub Actions standard runner cost):

Activity	Cost per run (baseline)	Cost per run (optimized)	Annual saving (50 PR/day, 250 days)
Suite execution	$0.13	$0.02	~$1 375 per suite

For organizations running multiple suites in parallel across many projects, the saving compounds quickly.

Two caveats

Find is not the only cost

The speedup applies to find time only. The full suite also contains network calls, server processing, browser rendering, mouse and keyboard events, and waits. A suite where finds dominate (visual regression, end-to-end smoke) sees larger wall-clock improvement than a suite waiting on slow back-ends.

Optimization rewards stability

Teams that regenerate patterns frequently see the chunk reset on each regeneration. The first run after regeneration pays the full-screen scan cost. The optimization rewards UI stability and frequent test execution — both characteristics of mature codebases.

Why this survives git clone and CI runners

The structural argument for embedding the position in the PNG itself, rather than in a sidecar file or a central index, is best expressed by considering the CI lifecycle.

A CI build typically starts from a clean state. A fresh container is provisioned, the repository is cloned from origin, dependencies are installed, the build runs, the tests run, the artifacts are collected, and the container is destroyed. There is no persistent state between builds. There is no shared file system. There is no cache that can be relied upon to be hot.

The oPLx chunk inside the PNG removes this discipline burden entirely. The chunk is part of the PNG file itself. When a developer runs the test suite locally and the chunk is updated, the modified PNG appears in git status automatically. Git tracks PNG files because they are committed in the project. The developer cannot commit “the update” without committing “the chunk inside the update”, because they are the same file.

This means that when a CI build clones the repository, it gets the PNG file with the chunk already inside. The first test run in CI is already in the fast path. The optimization is active from the first second. No warm-up phase is needed. No cache must be primed. The position metadata is in the file, and the file is in the repository, and the repository is in the clone.

A team running the suite locally, committing the regenerated PNGs, and pushing to CI is effectively pre-warming the CI cache through normal development activity. There is no separate “cache warm-up” step. The cache warms itself as a side effect of using the tool.

The broader pattern: PNG chunks as runtime context

The technique of embedding metadata in image files is not new. Adobe XMP, EXIF, IPTC, GIMP layer information, and many other systems use this mechanism. What is new, or at least underused, is the embedding of runtime state — not authoring metadata, not provenance information, but actual operational data that evolves as the file is used.

This pattern generalizes beyond visual automation:

Texture caches for graphics applications

A texture file could embed the time it was last loaded, the GPU memory pool it was allocated from, and the average frame time when it was active. A renderer could use this metadata to predict and pre-load textures.

Build artifacts for incremental compilation

A compiled object file could embed the hash of the source it was compiled from, the compiler version, and the optimization level. An incremental build tool could detect when recompilation is actually necessary.

Machine learning datasets

An image used in a vision model training set could embed its last classification result, the confidence score, and the model version. A cleaning tool could identify mislabeled samples without re-running the model.

Audit logs for compliance

A document image stored in a regulated workflow could embed a signed audit trail of the operations performed on it. The Ed25519 signing pattern from the OculiX MCP module would apply directly.

The common thread across all these examples is the same: keep operational state with the artifact, in a format the artifact’s primary tools will preserve, so that the state survives every distribution mechanism the artifact ever encounters.

What this doesn’t solve

A 6.5× speedup on the find is significant but not exhaustive. Several costs remain untouched by this optimization:

Cost component	Status	Approximate weight
Screen capture	Unchanged	20-40 ms per find
OpenCV setup	Unchanged	20-40 ms per find
Tesseract OCR loading	Unchanged	~300 ms per JVM cold start
OculiX framework init	Unchanged	~1.5 seconds per JVM cold start
JVM startup itself	Unchanged	~1-2 seconds per cold start

Reducing these other costs would require deeper structural changes: pre-built native images via GraalVM (the JVM startup), incremental OpenCV initialization (the OpenCV setup), lazy OCR loading (Tesseract), and possibly an alternative screen capture API on each platform. Each of these is a separate optimization project, none of which is in scope for the current change.

Closing: context engineering as a discipline

The lesson of this whole investigation is not “PNG chunks are cool” or “spatial memoization wins”. Both of those are true but uninteresting on their own.

The lesson is structural.

Finding these gaps requires a specific habit: benchmarking systems in the configuration users actually experience, not in the configuration that is convenient for developers. In a tight in-process loop, the OculiX find is fast. In a cold-start CI environment, it is slow. The same algorithm, the same code path, the same operating system. The only difference is whether the context built by the previous match has survived to inform the next match.

The fix is rarely a new algorithm. It is usually a new place to store something that already existed, in a form that can travel through the boundaries the user’s actual workflow imposes.

The PNG ancillary chunk happens to be a particularly elegant place to store visual automation context, because the artifact whose context we want to preserve is itself a PNG file, and the standard that defines PNG includes a mechanism specifically designed for this kind of metadata, and the tooling ecosystem has respected that mechanism for thirty years.

The result is an optimization that did not require inventing a new algorithm, did not require breaking any existing contract, did not require any external dependency, and is fully backward compatible with files that do not yet have the chunk. It just needed someone to measure the actual baseline, identify the gap, and close it.

For anyone maintaining a similar tool, my parting suggestion would be: go measure your cold-start performance against your warm-cache performance, in the same configuration your users actually run. If the gap is large, the optimization opportunity is probably not where you think it is. It is probably in the space between two of your existing components, in the form of a state that briefly exists and then disappears.

Repository: github.com/oculix-org/Oculix

Issue tracking the implementation of the persistent locator chunk: oculix-org/Oculix#353

Rebuilding DOMAutopsy in four hours, and the five things AI couldn't decide

Mon, 18 May 2026 08:00:00 GMT

I spent a Sunday afternoon rebuilding DOMAutopsy, the visual locator harvester I maintain, from a PySide6 desktop tool into a FastAPI server with a live web UI. The work took roughly four hours. Claude Code wrote most of the ~4 000 lines that landed.

That number, on its own, says nothing. It’s the kind of figure people quote on LinkedIn to sell a service. What it actually means depends entirely on which decisions you let the model take, and which decisions you keep for yourself.

Five decisions stuck with me from that session. None of them were technically hard. All of them would have produced a noticeably worse codebase if I had accepted the model’s first suggestion.

1. The system did not need RAG

The agent’s first proposal for capturing context across a long browsing session was to embed the page snapshots, store them in a vector index, and retrieve top-k chunks when the LLM needed to reason about the trajectory. It is a reasonable suggestion in 2026. It would also have added a dependency on a vector database, an embeddings model, and a retrieval pipeline that nobody on the user side asked for.

The real need was simpler. The agent observes the DOM live, the listener captures the selectors, the report aggregates everything at the end. There is no retrieval problem. There is a context augmentation problem, and it is solved by a flat structured payload passed to the LLM at each step.

The first architectural decision was to remove an entire layer that the model would have built without flinching.

2. CDP screencast, not noVNC

For the live web UI, two viable options surfaced: stream the Chromium headless instance over CDP screencast, or wrap it in a VNC server and serve it over noVNC. Both work. The model recommended noVNC because the literature it’s trained on shows it as the standard.

In practice, CDP screencast is lighter, runs in the same Playwright process, and skips an entire VNC server installation in the Docker image. The noVNC route would have added moving parts that nobody needs once Playwright already exposes everything via CDP.

This is the kind of call that takes thirty seconds of judgment and that the model has no way of making on its own. It optimizes for the average answer in its training set, not for the architecture you happen to be building today.

3. A seven-tier selector cascade beats a single LLM matcher

Several open source projects from the last twelve months propose to delegate selector generation entirely to an LLM at each interaction. The model handles the DOM, picks the most stable selector, returns the result. Elegant on paper.

DOMAutopsy uses a different approach: a deterministic seven-tier cascade in a 372-line JavaScript listener, with the LLM only intervening at cleanup time. Tier 1 is data-testid, id, name. Tier 7 is short CSS or text XPath. The LLM never sees a selector unless the cascade fails or produces ambiguous matches.

The reason is straightforward. A pure LLM-based matcher is non-deterministic, expensive at scale, and creates a hard dependency on a third party at every interaction. A static cascade is auditable, replayable offline, and cheap. The LLM adds value only where deterministic logic stops being effective.

This is a structural decision that no agent will make for you, because every example it has read describes the opposite approach.

4. Regex parser, not AST parser, for legacy scripts

DOMAutopsy can reverse-engineer existing test scripts (Katalon, Playwright, Cypress, Selenium) into structured tasks for the AI agent. The proposal from the model was to write a proper AST parser per language: Tree-sitter for Groovy, ts-morph for TypeScript, and so on.

For about 95 percent of real-world production test scripts, regex is enough. They are linear, mechanically generated, repetitive. AST parsing would have multiplied the implementation cost by ten and shifted the failure surface from “regex misses a syntax case” to “AST parser crashes on a malformed legacy file”.

I picked regex. I also added a clear note in the documentation: ~95 percent coverage, the rest goes to manual fallback. Honest constraint, manageable scope.

5. No SaaS lock-in, even when it would have been faster

At one point during the session, the model proposed to integrate a third-party hosted screenshot diff service to offload the visual comparison work. It would have shaved off some implementation time.

OculiX and DOMAutopsy both live in regulated environments. Banks, defense contractors, healthcare providers. Adding a third-party SaaS dependency without conscious decision would close doors I would later have to pry back open. I declined, the model accepted, the screenshot diff happens locally.

This is a non-technical call dressed up as a technical one. The model does not know that adding external-api.com to the data flow will make the procurement file twice as long for a CISO in a regulated sector. I do.

What AI actually does well, and what it doesn’t

In four hours, AI produced clean, working, mostly idiomatic code. The throughput is real. The output is not magic. It is the result of structured prompting, a clear mental model on the architect side, and a continuous flow of small judgments that close off bad branches before they grow.

The interesting question is not “can AI replace senior architects”. It is: which decisions are you still going to take yourself, and which are you ready to delegate. The five above happen to be calls I would refuse to delegate even with another twenty years of experience. They are domain-specific, context-aware, and they shape the rest of the project for years.

The rest, AI shipped it.

—

Repo: github.com/julienmerconsulting/DOMAutopsy