If provided, test-web batches & results will be partitioned in
multiple runs. Each run will execute the same set of tests, and
non-pass results will be stored in run1, run2, run6, etc directories.
This aligns our behaviour closer to other browsers, which
_mostly_ consider file scheme URLs as opaque. For test
purposes, allow overriding this behaviour with a commandline
flag.
I added the VERBOSITY_LEVEL_LOG_TEST_OUTPUT level for the case where I
run test-web with a filter for a single file, in which case I want to
just see the test output in the terminal. But for CI, we want to always
capture output for the uploaded test artifacts.
This patch changes this verbosity level to instead tee the output. So
we always create the results artifacts with all logs, and optionally
echo the received output.
If tests fail, let's just always create the results page regardless of
verbosity. This allows viewing logs afterwards, as well as uploading the
test-dumps collection in CI.
As a result, this removes the --dump-failed-ref-tests flag. It's a bit
overloaded with our new results format, and it would be awkward to keep
both separately working here.
We were previously using the absence of a results directory to decide
if we should disable live display. However, this was never the case -
the results directory would become unconditionally set in main(). This
caused all test-web output in CI to become hidden.
Instead, let's add a verbosity mode to display test output explicitly.
This is set for CI.
This also makes it much easier to debug single tests locally. The live
capture and results file combo is a bit overkill for running a single
test; we can now just add "-v" to the command line to see the output
directly in the terminal.
- Add --results-dir CLI flag to specify output directory
- Default to {tempdir}/test-web-results if not specified
- Capture stdout/stderr from all helper processes (WebContent,
RequestServer, ImageDecoder) to prevent output spam
- Save captured output to per-test files in results directory
- Save test diffs (expected vs actual) to results directory
- Generate HTML index of failed tests with links to diffs
- Display live-updating concurrent test status with progress bar
- Defer warning messages until after test run completes
This mode allows us to test the HTTP disk cache with two mechanisms:
1. If RequestServer is launched with --http-disk-cache-mode=testing, it
will cache requests with a X-Ladybird-Enable-Disk-Cache header.
2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response
header indicating how the response was handled by the cache. There is
no standard way for a web request to know what happened with respect
to the disk cache, so this fills that hole for testing.
This mode is not exposed to users.
This is used by tests to set the default time zone to UTC.
This is because certain tests create JavaScript Date objects, which are
in the current timezone.
We only supported headless clipboard management in test-web. So when WPT
tests the clipboard APIs, we would blindly try to access the Qt app,
which does not exist.
Note that the AppKit UI has no such restriction, as the NSPasteboard is
accessible even without a GUI.
Clipboard handling largely has nothing to do with the individual web
views. Rather, we interact with the system clipboard at the application
level. So let's move these implementations to the Application.
We have a test for localStorage that repeatedly adds new items until the
quota is exceeded:
`webstorage/storage_local_setitem_quotaexceedederr.window.html`.
This test becomes significantly slower when localStorage is backed by
SQL database. Let's disable database usage in test mode for now, as this
test is likely to be flaky on CI due to timeouts.
You would have to just know that you need to define the constructor with
this declaration. Let's allow subclasses to define constructors as they
see fit.
This is causing errors on the WPT runner, which does not have a display
output. To do this requires shuffling around the Main::Arguments struct,
as we now need access to it from overridden WebView::Application methods
after construction.
Now that headless mode is built into the main Ladybird executable, the
headless-browser's only purpose is to run tests. So let's move it to the
testing directory and rename it to test-web (a la test-js / test-wasm).
