This helps buddies from MI5, MI6, NSA etc. to run tests behind proxy.
Testing: Tested locally behind proxy. For some reason, I already set
`127.0.0.1` and `localhost` in `no_proxy`, but it does not work.
Fixes: #42321
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
All other browsers use a single configuration for their browser
invocations on WPT. Servo historically had two: servo and servodriver.
Now that we run WPT on Servo GitHub CI with Webdriver using the
servodriver, we can align our configuration with theirs.
The existing "servo" configuration is renamed to "servo_legacy" and
"servodriver" is then renamed to "servo". This way, we preserve the
"servo" product name as defined on wpt.fyi, but we do use its webdriver
configuration.
Since webdriver is not fully working yet for debugging purposes, we keep
the "servo_legacy" configuration now. In the future, once the debugging
story has improved, we can remove "servo_legacy".
All in all, this ensures that both on local, Servo GitHub CI and on
wpt.fyi we all use the exact same configuration. I tested this locally
by running the following test:
```
./mach test-wpt tests/wpt/tests/css/css-overflow/scrollbar-gutter-dynamic-004.html
```
This does times out with the servo binary and works with the servodriver
binary.
Running the servo_legacy configuration is done via the `--servo-legacy`
flag:
```
./mach test-wpt tests/wpt/mozilla/tests/mozilla/caption.html --servo-legacy
```
Fixes#40751
---------
Signed-off-by: Tim van der Lippe <tvanderlippe@gmail.com>
Nowadays Servo automatically loads system default proxy. It has enabled
my NSA colleagues to visit external networks without modifying transport
layer, whereas `test-wpt` has been failing after it.
We disable the proxy unless users know what they are doing.
Testing: Tested in different physical locations with different proxy
networks.
Fixes: #41440
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
The current user stylesheet use in our tests is a hack that hides real
issues like #40419. Any test that requires ahem.ttf needs to include the
stylesheet like any other webpage.
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
This is a more reasonable default as the detection of running one or
more tests is a bit inconsistent. It also makes the behavior of the
command a bit more predictable IMO.
Testing: This was tested manually as there is no tests for this
layer of the test runner
Fixes: #40407.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
In the review for #40369, the name of a command-line argument was
changed, but not the code that accessed the argument. This meant that
the unexpected log was never dumped, even when there are stable
unexpected results. This change fixes that.
Testing: This should ensure that the `stable-unexpected-results-linux`
artifact
is not empty when running WPT on CI.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This change switches the default test runner for WPT to be WebDriver,
enabling testdriver tests by default. In addition, it update results to
reflect the ones that you would expect when running with WebDriver.
While there are some failures that require more investigation, in
general the differences in results are fairly explicable.
Testing: This change modifies the way that tests are run and is thus
tested by all WPT-like tests.
Fixes: #34683
---------
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This change ensures that the filtered raw log file of WPT results only
contains stable unexpected results and uploads a resulting aggregated
log file as an artifact. This artifact can be used to generate new
baselines of *only* stable unexpected results, so is useful for updating
results given a CI run. A future change will add a `mach` command to do
this automatically.
Testing: The CI should run the code to produce this artifact for every
WPT run.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Instead of ignoring the `--headless` argument to the WPT test runner,
properly use it to turno on headless mode in Servo when its provided.
Additionally, when more than a single test is run, turn on headless mode
automatically to preserve existing behavior.
Although this change is for the legacy test driver, this will allow the
WebDriver test runner to run properly without providing the `--headless`
argument.
Testing: This modifies the way the test harness works, but the test
harness is untested.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This changes will introduce [flake8-annotations
(ANN)](https://docs.astral.sh/ruff/rules/#flake8-annotations-ann) for
python type annotation, this will make all thing related to function
strictly typed in python
This rule will start to affected this directory from now:
- /python -> Root directory
- /python/tidy
- /python/wpt
Testing: `./mach test-tidy`
Fixes: Not related to any issues
---------
Signed-off-by: Jerens Lensun <jerensslensun@gmail.com>
This ensures a consistent state when running tests that can otherwise
leave artifacts around (eg. localStorage/IndexedDB/cookies).
Testing: Existing test coverage is sufficient.
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
This is the first stage of adopting Pyrefly. It introduces the Python
folder and focuses on fixing issues around it.
Testing: *Describe how this pull request is tested or why it doesn't
require tests*
Fixes: *Link to an issue this pull requests fixes or remove this line if
there is no issue*
---------
Signed-off-by: Jerens Lensun <jerensslensun@gmail.com>
With this PR ServoHandler will store state of first suite and deal with
subsequent suites as they were run from `--retry-unexpected`, so it will
use results to mark unexpected from first run as flaky. Stats that are
used to display current running tests are still reset per suite. This
allows us to use `--retry-unexpected=1` for flake detection instead of
manual rerunning, which will help with proper subsuites support.
Testing: Manual CI run to ensure end results are still the same:
https://github.com/sagudev/servo/actions/runs/15886712204Fixes: #37319
---------
Signed-off-by: sagudev <16504129+sagudev@users.noreply.github.com>
#37021 exposed that our tests are trying to load a local file path for
Ahem.ttf, but this stops working when we implement the specified CORS
protections for our font requests. Luckily, the WPT suite already
provides a same-origin version of the same font, so we can just update
our CSS to use that font face instead.
Testing: Existing WPT coverage.
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
This is needed for subsuites to work with update-wpt.
Testing: We do not have tests for WPT scripts, but I manually tested
that update-wpt does not throw anymore.
Signed-off-by: sagudev <16504129+sagudev@users.noreply.github.com>
https://github.com/web-platform-tests/wpt/pull/50041 allows us to start
running the webdriver conformance tests in Servo, which will make it
easier for us to track regressions/improvements in our webdriver server
implementation.
---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes are part of #15274
- [x] There are tests for these changes
---------
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
We were already not compiling it and not running tests on it by default.
So it's simpler to just completely remove it.
Signed-off-by: Oriol Brufau <obrufau@igalia.com>
This is a speculative fix for #31845. Instead of sending all of the
output to the dashboard, send just the first 1024 characters. This value
can be adjusted in the future if it is too large or too small.
Fixes#31845.
* Fix `mach test-wpt` to make crash tests work
There are two issues related to crash tests:
1. test-wpt is unable to find existing crash tests even when
called with --test-types=crashtests. The fix here is to
add crashtests to the default test suite types to python/wpt/run.py
2. When running in headless mode, crashes in style threads
don't cause servo to crash because the logic in constellation.rs
currently calls handle_panic only when the top-level browsing
context id is some value. Since style pool threads are shared,
they always generate Panic messages with None as top-level
browsing context id.
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* Send bactrace to stderr and capture it in test runner
Servo's panic hook writes backtraces to stdout. This
patch changes it so they are written to stderr.
The crash test executor for servo in WPT grouping formatter
was also not capturing the output correctly for crashtests
as the log events were being aggregated based on thread name
which doesn't seem to match correctly in case of crashtests.
This patch also fixes the log grouping logic to be based on
test name.
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* crashtests: update expectations for layout 2020
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* crashtests: update expectations for layout 2013
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* remove outdated & intemittent test expectations
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
---------
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
There were some issues with the way that the `--release` and `--dev`
arguments were handled in mach commands.
- Not all commands accepted them in the same way. For instance `./mach
test-wpt` didn't really accept them at all.
- If you did not pass either of them, mach would try to guess which
build you meant. This guess was often quite surprising as it wasn't
printed and it depended on the state of the your target directory,
which is difficult to remember.
- The `dev` profile is colloquially called a "debug" profile and some
commands accepted `-d` or `--debug...` like arguments, but `--debug`
with `./mach run` meant run in a debugger. It was easy to mix this
up.
This change:
- Centralizes where build type argument processing happens. Now it the
same shared decorator in CommandBase.
- Uses a `BuildType` enum instead of passing around two different
booleans. This reduces the error checking for situations where both
are true.
- Be much less clever about guessing what build to use. Now if you
don't specify a build type, `--dev` is chosen. I think this behavior
matches cargo.
- Makes it so that `./mach test-wpt` accepts the exact same arguments
and has the same behavior as other commands. In addition, the suite
correct for `test-wpt` is removed. There are only two suites now and
it's quite unlikely that people will confuse WPT tests for rust unit
tests.
Currently, `./mach test-wpt` family of commands (`test-wpt`,
`test-wpt-android`, and `test-wpt-failure`) and `./mach update-wpt`
default to using the legacy-layout option `--layout-2013` unless
`--layout-2020` is specified.
Given that we are now using layout-2020 by default, this change updates
these commands to default to using the `--layout-2020` option instead of
`--layout-2013`.
This change moves all of Servo's WPT Python support scripts into one
directory as they were previously scattered throughout the directory
structure. This should allow more code reuse and make it easier to
understand how everything fits together.
The changes:
- `tests/wpt/update` → `python/wpt/importer`
- `etc/ci/upstream-wpt-changes/wptupstreamer` → `python/wpt/exporter`
- `etc/ci/upstream-wpt-changes/test.py` → `python/wpt/test.py`
- `etc/ci/upstream-wpt-changes/tests` → `python/wpt/tests`
- `tests/wpt/servowpt.py` →
- `python/wpt/update.py`
- `python/wpt/run.py`
- `tests/wpt/manifestupdate.py` → `python/wpt/manifestupdate.py`
This change also removes
- The ability to run the `update-wpt` and `test-wpt` commands without
using `mach`. These didn't work very well, because it was difficult
to get all of the wptrunner and mach dependencies installed outside
of the Python virtualenv. It's simpler if they are always run through
`mach`.
- The old WPT change upstreaming script that was no longer used.
