This script waits until fonts have loaded and waits for 2 animation
frames before signalling test completion. This is the same mechanism
already used for ref and crash tests.
Deferred warnings were originally intended to suppress output during
live display, ostensibly to avoid glitch scrolls.
Then, 1af74d1a7c added log capture to the
test-web process. Suddenly, deferred warnings became deadly because
they're flushed during a tiny window after the capture notifier has
stopped draining the tee pipe but before stderr is restored.
This caused a deadlock at exit. The fix is to remove this system and
call warnln directly because display integrity is now protected by
other means.
This prevents glitches where stale test status lines can bleed above
the live display list. Log this additional captured output so it can
be reviewed if a main or helper process failure is suspected.
Use files instead of buffers. Consolidate stderr and stdout into one
view. Break handling out into TestRunCapture and CaptureFile helper
classes. We will use them to log output from ALL processes in the next
commit.
When test-web runs inside Claude Code or Codex, per-test progress lines
are suppressed. These environments are non-interactive and the per-test
output (potentially thousands of lines) wastes the agent's context
window without adding value. The agent only needs the pass/fail summary
to decide next steps.
Quiet mode preserves: the "Running N tests..." line, the summary table,
non-passing test names, and the results URL.
This is a little more complicated than screenshot tests, because we deal
with two separate page loads, for the test and expectation pages. When
loading the expectation page, we clear the did_inject_js flag and also
store the URL so that we can compare that later.
Keep track of whether we've injected the JS, and avoid doing so again.
Also skip this entirely unless we're actually dealing with the test's
URL and not a subframe. This check is already done on dump tests.
The MutationObserver was not disconnected after firing, so any other
modifications to the document's class attribute after that point would
signal the test completing again. This is unlikely to happen in
practice as we shouldn't be running any JS after completing a test, but
it's harmless to disconnect it.
We had three separate timeout timer sites in `test-web`, which are now
deduplicated into a single one. This single timeout timer is started
_before_ we start loading `about:blank`, which potentially catches more
timeouts when a WebContent process is left in a broken state by a
previous test.
Previously, if you had a mistake in a test path, you'd get an unhelpful
error like this when running test-web:
> Runtime error: realpath: No such file or directory (errno=2)
This is quite confusing if other changes have been made, as the error
could have happened anywhere.
With this change, the same mistake now is first reported as:
> Unable to get real path for TestConfig.ini entry
> '/home/sam/ladybird/Tests/LibWeb/FLORB.html': realpath: No such file
> or directory (errno=2)
Then followed by the original error reported by LibMain. This makes it
immediately obvious what the issue is, and hopefully will save anyone
else going on a wild goose chase trying to find a build issue that isn't
there. 😅
When a path in TestConfig.ini ends in a /, treat it as a directory and
recursively enumerate all tests within it to the requisite collection
of paths.
We can immediately use this to drop a bunch of lines from the config
that specify individual IndexedDB tests should use the HTTP server.
If provided, test-web batches & results will be partitioned in
multiple runs. Each run will execute the same set of tests, and
non-pass results will be stored in run1, run2, run6, etc directories.
Previously, --rebaseline would unconditionally overwrite expected PNGs
before comparing, causing every screenshot test expectation to be
regenerated even when the actual screenshot already matched. Restructure
to load and compare first, only writing the new expectation on mismatch
or when the expected file doesn't exist yet.
Instead of rendering a reference HTML page that wraps an <img> tag
pointing to a PNG, Screenshot tests now load the expected PNG directly
from disk and compare it against the rendered screenshot. This
eliminates the indirection of loading and rendering a second page just
to display a static image.
This also means --rebaseline now works for Screenshot tests, generating
the expected PNG automatically instead of requiring manual screenshot
capture and placement.
Changes:
- Add TestMode::Screenshot with its own collector and runner
- Move PNGs from Screenshot/images/ to Screenshot/expected/ with
normalized names matching input filenames
- Remove all 92 reference HTML wrapper files and the images/
directory
- Remove <link rel="match"> from all 94 Screenshot input HTML
files
- Update add_libweb_test.py Screenshot boilerplate accordingly
- Add Screenshot mode to results viewer image comparison tabs
If the difference between the expected and actual test images was small,
our in-browser diff tool would often fail to highlight differing pixels.
Replace this by generating a new diff PNG that is layered as follows:
1. 50%/50% blend of the actual and expected images
2. 80% blend with white / rgb(255, 255, 255)
3. Differing pixels are highlighted in red / rgb(255, 0, 0)
Many web features work differently when loaded from file:// URLs
as web features such as origin checks no longer hold. For example,
document.cookies is disabled when a file URL is loaded. It also
means that many WPT tests when imported do not work as intended.
While it would be nice to load almost all tests from an HTTP server
this makes running tests significantly slower. Furthermore, loading
tests over a file:// URL is sufficient for our needs in most
scenarios anyhow.
To support both usecases, add the ability to configure a
specific test in TestConfig.ini to be loaded over an HTTP server
instead of its file:// URL.
Commit 84db5d8c1c introduced the ability
to load tests over an http:// URL instead of a file:// URL. Each time
this happens, we switch to a new WebContent process due to site
isolation. Our WebContent output capture was not handling this.
For some reason, this was causing a wide array of test failures and
timeouts. Often, the failures were accompanied by the content of the
files loaded over HTTP being dumped to stdout. It's not quite clear
what was going on here.
I added the VERBOSITY_LEVEL_LOG_TEST_OUTPUT level for the case where I
run test-web with a filter for a single file, in which case I want to
just see the test output in the terminal. But for CI, we want to always
capture output for the uploaded test artifacts.
This patch changes this verbosity level to instead tee the output. So
we always create the results artifacts with all logs, and optionally
echo the received output.
When dumping a GC graph, we now write the output as a .js file
containing `var GC_GRAPH_DUMP = <json>;` instead of raw JSON.
This allows gc-heap-explorer.html to load the dump via a
dynamically created <script> element, avoiding CORS restrictions
that prevent file:// pages from fetching other file:// URLs.
After dumping, both the browser and test-web print a clickable
file:// URL that opens the heap explorer with the dump pre-loaded.
The heap explorer's drag-and-drop file picker also accepts both
the new .js format and plain .json files.
If tests fail, let's just always create the results page regardless of
verbosity. This allows viewing logs afterwards, as well as uploading the
test-dumps collection in CI.
As a result, this removes the --dump-failed-ref-tests flag. It's a bit
overloaded with our new results format, and it would be awkward to keep
both separately working here.
We were previously using the absence of a results directory to decide
if we should disable live display. However, this was never the case -
the results directory would become unconditionally set in main(). This
caused all test-web output in CI to become hidden.
Instead, let's add a verbosity mode to display test output explicitly.
This is set for CI.
This also makes it much easier to debug single tests locally. The live
capture and results file combo is a bit overkill for running a single
test; we can now just add "-v" to the command line to see the output
directly in the terminal.
This fixes a regression introduced in bf77aeb3d which added support for
WPT variant meta tags.
The problem occurs when a text test has no corresponding expectation
file. The test should fail, but instead it passes. This happens because
of the following execution flow:
1. Test loads and on_load_finish is called
2. internals.loadTestVariants() is executed (for all text tests)
3. The test JS calls signalTestIsDone(), setting did_finish_test = true
4. The variant metadata callback fires with an empty array (no variants)
5. In on_test_variant_metadata, since did_finish_test is true, it
directly calls
view.on_test_complete({ test_index, TestResult::Pass })
The bug is in step 5: the code bypasses handle_completed_test() which is
where the expectation file comparison happens. Instead of calling
on_test_complete() (the lambda that invokes handle_completed_test), it
directly returns Pass.
When tests are being collected, the paths get canonicalized via the
FileSystem::real_path() call. This means on Windows the paths are using
`\` as the directory separator.
Tests get removed by performing string-based matching on a hardcoded
set of support file globs and an optional user defined glob. Even if we
normalized these globs to use `\` instead of `/` on Windows, they still
wouldn't match because AK::StringUtils::matches() uses `\` to escape
metacharacters. For example, `my\relative\windows\path` will not match
`my\*\path` because the mask is interpreted as `"my*\path"` instead.
So we normalize paths to use POSIX separators to support the existing
glob matching infrastructure. This allows Windows to filter out support
files properly which prevents us from crashing when Ref tests try to
load one of those files. It also allows Windows to use `/`-based paths
for the -f argument.
Use to_truncated_seconds() instead of to_seconds() for the per-view
duration display. The latter rounds up, causing "1s" to appear almost
immediately when a test starts. Now "1s" only appears after a full
second has elapsed.
Previously, when a test's expectation file didn't exist, test-web would
return an error immediately without generating diff output files. This
made it difficult to see what the actual test output was.
Now, when an expectation file is missing, test-web continues to generate
the diff files (showing all output as additions) so users can inspect
the actual output and create expectation files if needed.
Also defer the "Failed opening" warning to avoid cluttering the live
display output.
Add support for WPT test variants, which allow a single test file to be
run multiple times with different URL query parameters. Tests declare
variants using `<meta name="variant" content="?param=value">` tags.
When test-web encounters a test with variants, it expands that test into
multiple runs, each with its own expectation file using the naming
convention `testname@variant.txt` (e.g., `test@run_type=uri.txt`).
Implementation details:
- WebContent observes variant meta tags and communicates them to the
test runner via a new `did_receive_test_variant_metadata` IPC call
- test-web dynamically expands tests with variants during execution,
waking idle views after each test completion to pick up new work
- Use index-based test tracking to avoid dangling references when the
test vector grows during variant expansion
- Introduce TestRunContext to group test run state, and store a static
pointer to it for signal handler access
This enables proper testing of WPT tests that use variants, such as the
html5lib parsing tests (which test uri, write, and write_single modes)
and the editing/bold tests (which split across multiple ranges).
Two fixes for crash handling:
1. Re-setup output capture after WebContent respawns. When WebContent
crashes, a new process is spawned with new stdout/stderr pipes.
We need to set up new notifiers to drain them, otherwise the new
WebContent blocks on write() when its pipe buffer fills up.
2. Disconnect child view crash handlers between tests. Child views
(from window.open, iframes, etc.) persist after a test completes.
If they crash later, we don't want that to affect subsequent tests.
The timeout timer for ref tests was stored in a local variable that
went out of scope when run_ref_test() returned, causing the timer to
be destroyed before it could fire.
Store the timer in test.timeout_timer (like dump tests already do)
so it persists until the test completes.
When a WebContent process crashes, its stdout/stderr pipes will signal
POLLHUP. The output capture notifiers were not handling this case,
causing the event loop to spin at 100% CPU as poll() kept returning
immediately with POLLHUP set.
Fix by disabling the notifiers when read() returns 0 or an error.
