Skip to content

Tighten screenshot comparison coverage#1730

Open
matthargett wants to merge 4 commits into
BabylonJS:masterfrom
rebeckerspecialties:tighten-screenshot-thresholds
Open

Tighten screenshot comparison coverage#1730
matthargett wants to merge 4 commits into
BabylonJS:masterfrom
rebeckerspecialties:tighten-screenshot-thresholds

Conversation

@matthargett

@matthargett matthargett commented Jun 3, 2026

Copy link
Copy Markdown

Summary

  • tighten visual comparison budgets for GUI3D SpherePanel, GUI Slate, and GUI Near Menu
  • add Standard/PBR texture-repetition coverage with native-compatible RGBA references
  • use a synchronous GUI Slate playground without test-owned readiness or render-loop warmup
  • keep the GUI gradient cases quarantined with precise bgfx Canvas limitation reasons

Why

The default 2.5% screenshot budget permits roughly 6,000 differing pixels in a 600x400 image. That is enough to hide missing or materially underdrawn controls and compact material differences. The tighter per-test budgets preserve normal anti-aliasing/color tolerance while making those regressions actionable.

The three GUI gradient cases are not tolerance problems on bgfx. Forced Metal/bgfx runs show missing Canvas semantics for gradient stroke styles, two-circle radial geometry, and endpoint-space linear mapping, so they remain excluded rather than being marked passing with loose thresholds.

Dependencies

GUI Slate requires BabylonJS/Babylon.js#18709 to be present in BabylonNative before this PR merges. That fix coerces fractional native dynamic-texture dimensions before allocation.

This PR does not depend on #1762. The replacement Slate playground is synchronous and its configured 60 validation frames produce the same capture with and without the readiness pump.

Validation

  • config JSON parse and git diff --check
  • GUI Slate, Metal/bgfx: 1,708 of 240,000 pixels (0.712%) at threshold 25
  • GUI Slate, NativeWebGPU: 0 pixels over threshold 25
  • GUI Slate A/B with and without Preserve animation frames during screenshot readiness renders #1762: byte-identical captures
  • Texture Repetition Standard, NativeWebGPU: 2 pixels over threshold 25
  • Texture Repetition PBR, NativeWebGPU: 761 pixels (0.317%) over threshold 25

Copilot AI review requested due to automatic review settings June 3, 2026 20:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates the Playground visual regression configuration to better handle pixel-diff variability and to add new texture repetition baseline tests.

Changes:

  • Added per-test pixel comparison tuning via threshold and/or errorRatio for several existing tests.
  • Added two new “Texture Repetition” playground entries with reference images and engine exclusions.

Comment thread Apps/Playground/Scripts/config.json Outdated
Comment thread Apps/Playground/Scripts/config.json
Comment thread Apps/Playground/Scripts/config.json

@bkaradzic-microsoft bkaradzic-microsoft left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tightening the screenshot tolerances — the per-test threshold/errorRatio approach is the right direction. I ran the changed/added tests locally against a Playground build (D3D11 + OpenGL) and hit blocking issues on the re-enabled and new tests. Two are structural and independent of Babylon version; a few others I'd like you to confirm pass in your CI, since my local build may differ from your base.

Blocking:

  1. New Texture Repetition reference images are 24-bit RGB, but every existing reference is 32-bit RGBA. The harness compare() requires equal decoded-buffer lengths, so both new tests throw an uncaught Reference data length (720000) must match render data length (960000) and exit 1 — they can never pass on Babylon Native. Please re-export both PNGs as RGBA (ideally captured from Native's own output rather than the web playground).
  2. Un-excluding OpenGL for GUI Near Menu re-introduces a hard crash. On the OpenGL build it hits BGFX FATAL: Failed to compile shader ... '=' : cannot convert from 'highp float' to 'bool' (renderer_gl.cpp:6956) — exactly the failure the reason you removed described. This aborts the entire OpenGL test run. The OpenGL exclusion should stay (e.g. excludedGraphicsApis: ["OpenGL"]) until the GUI fragment-shader codegen issue is fixed.

Please verify (may be environment-specific on my end):

  1. Re-enabled GUI Gradient Radial / Linear with transparency don't crash/hang here (good — that reason is stale), but they diff ~60k/~102k pixels, far above errorRatio: 1.0 (~1%). GUI Gradient Linear throws TypeError: A string was expected, and GUI Slate fails to evaluate with The data size does not match width, height, and format. Can you confirm these actually pass in your run?

Minor: excludedEngines and dependsOn on the new entries aren't implemented by validation_native.js (only excludedGraphicsApis, onlyVisual, excludeFromAutomaticTesting are honored) — they'll be silently ignored, giving a false sense of gating.

Comment thread Apps/Playground/Scripts/config.json Outdated
Comment thread Apps/Playground/Scripts/config.json Outdated
Comment thread Apps/Playground/Scripts/config.json
Comment thread Apps/Playground/Scripts/config.json Outdated
Comment thread Apps/Playground/Scripts/config.json Outdated
@bkaradzic-microsoft

Copy link
Copy Markdown
Member

Heads-up on CI: this branch is currently ~38 commits behind master (last pushed 2026-06-03), and there's no GitHub Actions check-suite on the head commit — so there's no pending workflow run for a maintainer to approve, which is why CI isn't showing up here.

Could you please rebase onto master and push? That synchronize event will create a fresh CI run that we can approve, and it'll also pick up the current workflow set. Thanks!

Store the texture repetition references as 32-bit RGBA so their decoded buffers match Native screenshot output.

Keep GUI Near Menu excluded from OpenGL until its bgfx shader compile failure is fixed, and remove browser-only metadata that validation_native.js does not implement.
@matthargett
matthargett force-pushed the tighten-screenshot-thresholds branch from b85623e to 4d42eb3 Compare July 18, 2026 00:26
@matthargett

Copy link
Copy Markdown
Author

Rebased onto current master (fcfda740) and force-pushed. I also removed the unsupported excludedEngines / dependsOn fields from the two native test entries. The two GUI verification threads remain open while I reconcile the reported runtime failures.

Replace the deleted GUI Slate snippet with its readiness-aware successor. The replacement preserves the same HolographicSlate scene and is pixel-identical to the existing golden.\n\nUse a 0.75% error ratio based on three stable Metal/bgfx captures at 1,708 of 240,000 differing pixels with threshold 25. This remains substantially tighter than the default 2.5% allowance while accounting for backend edge-antialiasing drift. NativeWebGPU has zero pixels over threshold 25.\n\nThe replacement also exposed fractional dynamic texture dimensions in ThinNativeEngine; BabylonJS/Babylon.js#18709 contains the separate runtime fix needed for bgfx backends to reach screenshot comparison.
Use a synchronous HolographicSlate playground without a private readiness or render-loop workaround. The configured 60 validation frames produce the same image with and without the separate readiness-pump change, keeping these PRs independent.\n\nKeep the GUI gradient cases excluded on bgfx and document the concrete Canvas API gaps: gradient stroke styles, two-circle radial geometry, and endpoint-space linear mapping. These are rendering limitations rather than tolerances that should be hidden with a permissive error ratio.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants