For every ref tests actual and expected screenshots are taken. These
screenshots are only needed while the individual test executes. However,
they are never freed during the run, leading to a continuous increase in
memory usage of the test runner while executing ref tests.
With the number of ref tests growing, this currently amounts to nearly 3
GB of uncompressed bitmap data being held in memory. Lets avoid that by
clearing the screenshot data at the end of each test. With this change
applied, the memory usage of test-web stays stable and below 100 MB for
the entire test run.
When a test is active in a test-web view, show the relative path to the
test instead of the view's URL. This gives a better starting point for
debugging than whatever the last loaded URL happened to be.
If no test is active, we still show the view's URL.
The BUILD_RPATH/INSTALL_RPATH CMake infrastructure is not supported
on Windows, but we want to ensure Ladybird executables are runnable
after the build phase so there can be an efficient dev loop.
lagom_copy_runtime_dlls() can be used by executable targets so all
their dependent dlls are copied to their output directory in their
post build step.
We occasionally (frequently) time out in CI. If ctest triggers this
timeout, it sends a SIGSTOP followed by a SIGKILL. Since we want to
gracefully exit the test runner, ask ctest to send a SIGTERM instead.
This should cause active test status reports to show up in CI.
If you interrupt the test runner (Ctrl+C, SIGINT) or if the test runner
is gracefully being terminated (SIGTERM), it now reports the current
status of all the spawned views with their URL and, if an active test is
still being run, the time since the start of the test.
Hopefully this will help gain insight into which tests are hanging.
The test logs tend to get a bit mixed together, so this makes it
possible to identify which values correspond to which test when multiple
are failing at once.
With the newly supported fuzzy matching in our test-web runner, we can
now define the expected maximum color channel and pixel count errors per
failing test and set a baseline they should not exceed.
The figures I added to these tests all come from my macOS M4 machine.
Most discrepancies seem to come from color calculations being slightly
off.
WPT reference tests can add metadata to tests to instruct the test
runner how to interpret the results. Because of this, it is not enough
to have an action that starts loading the (mis)match reference: we need
the test runner to receive the metadata so it can act accordingly.
This sets our test runner up for potentially supporting multiple
(mis)match references, and fuzzy rendering matches - the latter will be
implemented in the following commit.
This produces more granural information on the actual pixel errors
present between two bitmaps. It now also asserts that both bitmaps are
the same size, since we should never compare two differently sized
bitmaps anyway.
Of the available 71 screenshot tests that we have, 42 fail on macOS
arm64. Let's make it possible to skip those and run the remaining
succeeding screenshot tests on arm64 anyway.
When a test has a `reftest-wait` class, we now wait for fonts to load
and then for 2 `requestAnimationFrame` callbacks. This matches the
behavior of the WPT runner and allows more imported WPT tests to work
correctly.
We have a test for localStorage that repeatedly adds new items until the
quota is exceeded:
`webstorage/storage_local_setitem_quotaexceedederr.window.html`.
This test becomes significantly slower when localStorage is backed by
SQL database. Let's disable database usage in test mode for now, as this
test is likely to be flaky on CI due to timeouts.
You would have to just know that you need to define the constructor with
this declaration. Let's allow subclasses to define constructors as they
see fit.
This is causing errors on the WPT runner, which does not have a display
output. To do this requires shuffling around the Main::Arguments struct,
as we now need access to it from overridden WebView::Application methods
after construction.
Now that headless mode is built into the main Ladybird executable, the
headless-browser's only purpose is to run tests. So let's move it to the
testing directory and rename it to test-web (a la test-js / test-wasm).
