This helps buddies from MI5, MI6, NSA etc. to run tests behind proxy.
Testing: Tested locally behind proxy. For some reason, I already set
`127.0.0.1` and `localhost` in `no_proxy`, but it does not work.
Fixes: #42321
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
All other browsers use a single configuration for their browser
invocations on WPT. Servo historically had two: servo and servodriver.
Now that we run WPT on Servo GitHub CI with Webdriver using the
servodriver, we can align our configuration with theirs.
The existing "servo" configuration is renamed to "servo_legacy" and
"servodriver" is then renamed to "servo". This way, we preserve the
"servo" product name as defined on wpt.fyi, but we do use its webdriver
configuration.
Since webdriver is not fully working yet for debugging purposes, we keep
the "servo_legacy" configuration now. In the future, once the debugging
story has improved, we can remove "servo_legacy".
All in all, this ensures that both on local, Servo GitHub CI and on
wpt.fyi we all use the exact same configuration. I tested this locally
by running the following test:
```
./mach test-wpt tests/wpt/tests/css/css-overflow/scrollbar-gutter-dynamic-004.html
```
This does times out with the servo binary and works with the servodriver
binary.
Running the servo_legacy configuration is done via the `--servo-legacy`
flag:
```
./mach test-wpt tests/wpt/mozilla/tests/mozilla/caption.html --servo-legacy
```
Fixes#40751
---------
Signed-off-by: Tim van der Lippe <tvanderlippe@gmail.com>
Instead of relying on the contents of the already-parsed arguments,
which may have tried to convert URL arguments to paths, parse URLs from
their original place in `sys.argv`. This is a workaround to the fact
that we extend the internal WPT update command to accept URLs of GitHub
Actions. This is a custom Servo extension that has no equivalent to
the WPT upstream command.
Testing: There aren't really tests for this level of the Servo command
line tools.
Fixes: #41728
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Nowadays Servo automatically loads system default proxy. It has enabled
my NSA colleagues to visit external networks without modifying transport
layer, whereas `test-wpt` has been failing after it.
We disable the proxy unless users know what they are doing.
Testing: Tested in different physical locations with different proxy
networks.
Fixes: #41440
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
This just fixes a typo:
`download_run_resultsa_and_then_run_update` ->
`download_run_results_and_then_run_update`
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
This change allows `./mach update-wpt` to accept a URL to a GitHub
Action run. If a URL is passed, it will attempt to download the stable
unexpected results from the run and update the expected test results.
Note that this currently requires having the `gh` command-line tool
installed and authenticated. Although you can download artifacts without
logging in via the web interface, doing this via the API requires an
access token.
Testing: This change adds a unit test for the regex that matches GitHub
Action URLs.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Properly handle empty commit messages when processing commits during WPT
export. We shouldn't be landing commits with empty messages into Servo,
but sometimes when a PR is in process, the body is empty. In those
cases, this change avoids an error during job execution.
Signed-off-by: chenura999 <chenuraoshada396@gmail.com>
The current user stylesheet use in our tests is a hack that hides real
issues like #40419. Any test that requires ahem.ttf needs to include the
stylesheet like any other webpage.
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
This is a more reasonable default as the detection of running one or
more tests is a bit inconsistent. It also makes the behavior of the
command a bit more predictable IMO.
Testing: This was tested manually as there is no tests for this
layer of the test runner
Fixes: #40407.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
In the review for #40369, the name of a command-line argument was
changed, but not the code that accessed the argument. This meant that
the unexpected log was never dumped, even when there are stable
unexpected results. This change fixes that.
Testing: This should ensure that the `stable-unexpected-results-linux`
artifact
is not empty when running WPT on CI.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This change switches the default test runner for WPT to be WebDriver,
enabling testdriver tests by default. In addition, it update results to
reflect the ones that you would expect when running with WebDriver.
While there are some failures that require more investigation, in
general the differences in results are fairly explicable.
Testing: This change modifies the way that tests are run and is thus
tested by all WPT-like tests.
Fixes: #34683
---------
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This change ensures that the filtered raw log file of WPT results only
contains stable unexpected results and uploads a resulting aggregated
log file as an artifact. This artifact can be used to generate new
baselines of *only* stable unexpected results, so is useful for updating
results given a CI run. A future change will add a `mach` command to do
this automatically.
Testing: The CI should run the code to produce this artifact for every
WPT run.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Instead of ignoring the `--headless` argument to the WPT test runner,
properly use it to turno on headless mode in Servo when its provided.
Additionally, when more than a single test is run, turn on headless mode
automatically to preserve existing behavior.
Although this change is for the legacy test driver, this will allow the
WebDriver test runner to run properly without providing the `--headless`
argument.
Testing: This modifies the way the test harness works, but the test
harness is untested.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
We were capturing test output (such as stack traces) by using the "test"
key in the data provided to the `process_output` log handler. This
doesn't work for WebDriver mode because the test runner runs in another
process entirely and it may reuse Servo executable instances for tests.
This change maps browser process output to tests using the "browser_pid"
key that is often provided while in WebDriver mode.
Testing: This modifies the test harness, so the tests are running tests.
Fixes: #39176
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Architecture:
```
Desktop (Test Controller) OHOS Device (Test Target)
┌─────────────────────────┐ ┌─────────────────────────┐
│ WPT Server (port 8000) │ │ Servo Browser │
│ Test Runner Script │---->│ WebDriver Server (7000) │
│ HDC Port Forwarding │ │ Test Execution │
└─────────────────────────┘ └─────────────────────────┘
```
After the test is finished, the script will parse the results and print
them in a readable format.
Tried to handle as many errors as possible and find workarounds for each
error to ensure the testing can be completed, or at least provide
comprehensive logs or information to identify exactly where the problem
is. Note that the used ports are just for testing; you can change them
to any other available ports, but make sure that the ports are
consistent in the script and given commands.
To run a WPT test on an OHOS device, you need to:
1. Connect OHOS device to the desktop via a cable (didn't try any other
way of communication)
2. Build and deploy servo with the changes in this PR using
[servoDemo](https://github.com/jschwe/ServoDemo). You can find there the
instructions to build and deploy servo to OHOS device.
3. While deploying servo to OHOS you need to ensure WebDriver is enabled
with the argument --webdriver=7000
4. Ensure OHOS SDK with HDC in PATH
5. Start WPT server on the desktop on a different terminal in servo
directory: ```bash python -m wpt serve --port 8000 ```
6. Update desktop IP in test scripts:
```python
desktop_ip = "192.168.1.100" # Your desktop's IP
```
You can find your desktop IP with:
```bash
# Windows
ipconfig | findstr "IPv4"
# macOS/Linux
ifconfig | grep "inet "
```
script can be modified to detect the desktop's IP automatically ...
later.
7. Run tests using the new mach command:
```bash
./mach test-ohos-wpt \
--test <test relative path> \
--webdriver-port 7000 \
--wpt-server-port 8000 \
--verbose
```
The script will:
1. Set up HDC port forwarding and reverse port forwarding for WPT
automatically
2. Connect to WebDriver server on the device
3. Navigate to the test URL
4. Wait for test completion
5. Show test results
Troubleshooting common Issues and Solutions:
1. HDC command not found:
- Install OHOS SDK and add HDC to PATH
- Verify: `hdc --version`
2. Failed to connect to WebDriver:
- Ensure Servo is running with `--webdriver=7000` argument
- Check device connection: `hdc list targets`
- Verify port forwarding: `hdc fport ls`
- Test WebDriver directly: `curl http://localhost:7000/status`
3. Failed to navigate to test URL:
- Update `desktop_ip` in the script
- Ensure both devices are on same network or connected via cable
- Test connectivity: ping from device to desktop
4. Test timeouts:
- Increase timeout in script (default: 30 seconds)
- Check if test requires specific dependencies
- Verify WPT server is serving the test file
---------
Signed-off-by: abdelrahman1234567 <abdelrahman.hossameldin.awadalla@huawei.com>
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
Co-authored-by: Euclid Ye <yezhizhenjiakang@gmail.com>
This changes will introduce [flake8-annotations
(ANN)](https://docs.astral.sh/ruff/rules/#flake8-annotations-ann) for
python type annotation, this will make all thing related to function
strictly typed in python
This rule will start to affected this directory from now:
- /python -> Root directory
- /python/tidy
- /python/wpt
Testing: `./mach test-tidy`
Fixes: Not related to any issues
---------
Signed-off-by: Jerens Lensun <jerensslensun@gmail.com>
This ensures a consistent state when running tests that can otherwise
leave artifacts around (eg. localStorage/IndexedDB/cookies).
Testing: Existing test coverage is sufficient.
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
This is the first stage of adopting Pyrefly. It introduces the Python
folder and focuses on fixing issues around it.
Testing: *Describe how this pull request is tested or why it doesn't
require tests*
Fixes: *Link to an issue this pull requests fixes or remove this line if
there is no issue*
---------
Signed-off-by: Jerens Lensun <jerensslensun@gmail.com>
The WPT exporter currently assumes that removing a label from a github
PR will never fail. This is obviously false and the result is a silent
failure. By moving the operation inside the existing try block we ensure
that a comment will be posted if any part of the merge step fails.
Testing: Untested; did not feel worth the time investment to determine
if the existing test harness can mock this particular situation.
Fixes: #37914
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
With this PR ServoHandler will store state of first suite and deal with
subsequent suites as they were run from `--retry-unexpected`, so it will
use results to mark unexpected from first run as flaky. Stats that are
used to display current running tests are still reset per suite. This
allows us to use `--retry-unexpected=1` for flake detection instead of
manual rerunning, which will help with proper subsuites support.
Testing: Manual CI run to ensure end results are still the same:
https://github.com/sagudev/servo/actions/runs/15886712204Fixes: #37319
---------
Signed-off-by: sagudev <16504129+sagudev@users.noreply.github.com>
#37021 exposed that our tests are trying to load a local file path for
Ahem.ttf, but this stops working when we implement the specified CORS
protections for our font requests. Luckily, the WPT suite already
provides a same-origin version of the same font, so we can just update
our CSS to use that font face instead.
Testing: Existing WPT coverage.
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
This is needed for subsuites to work with update-wpt.
Testing: We do not have tests for WPT scripts, but I manually tested
that update-wpt does not throw anymore.
Signed-off-by: sagudev <16504129+sagudev@users.noreply.github.com>
https://github.com/web-platform-tests/wpt/pull/50041 allows us to start
running the webdriver conformance tests in Servo, which will make it
easier for us to track regressions/improvements in our webdriver server
implementation.
---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes are part of #15274
- [x] There are tests for these changes
---------
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
We were already not compiling it and not running tests on it by default.
So it's simpler to just completely remove it.
Signed-off-by: Oriol Brufau <obrufau@igalia.com>
The previous code was simply reporting the number of categories,
instead of the sum of the number of expected tests in each category.
Signed-off-by: Simon Wülker <simon.wuelker@arcor.de>
Before we appended all subdirs of unused_dir to unused_dirs, which caused errors on removing because root dir was already removed.
Fix#35026, tested locally.
Signed-off-by: Samson <16504129+sagudev@users.noreply.github.com>
wptrunner internally derives the path to the MANIFEST.json file from the
`metadata_path` passed via `test_paths`. The current logic overrides
only the `metadata_path`, leaving the manifest_path pointing at the
wrong (non-legacy) layout's MANIFEST.json.
In #34436 we observed that the recent WPT imports create a transient
`.cache` diretory that wptrunner logic uses for optimization. This has
not been observed until #34436 because a [recent bump in MANIFEST.json's
schema version][1] triggered the creation of the cache. Because of the
above issue with incorrect path and the fact that we *first* trigger
`mach update-wpt` for legacy layout during WPT import, the MANIFEST.json
of non-legacy layout gets incorrectly migrated during the invocation of
`update-wpt` for legacy layout but the cache is still created under
legacy-layout's path as it is not based on `manifest_path`. The
subsequent invocation of `mach update-wpt` for non-legacy finds the
MANIFEST.json already migrated so the `.cache` directory is not
constucted.
This change simply replaces the whole object using the wptrunner's
`TestRoot` class constructor so that all derived paths are calculated
correctly. We also add the `.cache` folders to gitignore as it seems
like they are expected to be created during such version migrations.
[1]: https://github.com/web-platform-tests/wpt/pull/49406
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
The GitHub search API is a bit sensitive. There isn't a great way to
search for the repository organization and the branch name when looking
for open PRs. Instead use the bot username as the author name, which
should likely have been having before. This fixes the WPT export job.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This switches the WPT export script to use Servo's fork the `wpt`
upstream repository. The reasoning is that Servo developers's can fix
issues with branches there. This isn't possible because no one has
permissions to the servo-wpt-sync fork of the repository.
I have given `servo-wpt-sync` write persmissions to `servo/wpt`.
This also updates the tests a bit to reflect the new default repository.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This is a speculative fix for #31845. Instead of sending all of the
output to the dashboard, send just the first 1024 characters. This value
can be adjusted in the future if it is too large or too small.
Fixes#31845.
1. Make the tidy output easier to follow
2. Integrate the WPT manifest cleanliness step into tidy
itself and don't run it if nothing has changed in the WPT
directory.
3. Fix an issue where Python test requirements were not installed,
which could cause issues with some modules not being found.
Fixes#30002.
Upstream wpt script has bug where 'product' defaults to
'firefox' and this causes import of uninstalled python
modules specific to firefox runners.
Fixes#30452
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
