In a fresh container skipping the apt-update will cause only 4 packages
to be installed and the rest filtered. Also add a debug print to inform
the user if packages were skipped.
Testing: Manually tested
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
`mach bootstrap` no longer installs Wireshark, as it isn't useful for
people not working directly on DevTools support. The linux runner that
runs `mach test-scripts` now installs it manually, skipping the security
prompt.
Testing: Manual test of `mach test-scripts`
Fixes: #40503
Signed-off-by: eri <eri@igalia.com>
Rename the `--scan` argument to `-w/--write-file` and the `--use`
argument to `-r/--read-file`. This is more aligned with `tshark`'s
syntax, and I think it is a more intuitive naming scheme.
Remove the `--filter` and `--range` arguments. They are very easily
replaced by more powerful tools like `grep` and `jq`. It seems
unnecessary to have them in this script (specially when the most useful
thing it does is exporting the capture as NDJSON for other tools to
process). This fixes an issue with the last message not being exported.
Change the default port to `6080`. This matches the current information
[in the
book](https://book.servo.org/hacking/using-devtools.html#connecting-to-servo)
on how to run Servo with DevTools enabled.
Testing: Checked with `math test-scripts`
---------
Signed-off-by: eri <eri@igalia.com>
An implementation detail of `./mach try` is that it creates a git commit
containing information about the try configuration. Adding `--no-verify`
skips running git pre-commit hooks, which could cause the commit to
fail. This is useful for draft / work-in-progress branches, where you
quickly want to commit something and test CI works, before fixing lints
/ formatting etc.
Testing: Manually tested with `./mach try`.
Fixes: #40475
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
This is a more reasonable default as the detection of running one or
more tests is a bit inconsistent. It also makes the behavior of the
command a bit more predictable IMO.
Testing: This was tested manually as there is no tests for this
layer of the test runner
Fixes: #40407.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Add a `--coverage` flag to `./mach build` and a `./mach coverage-report`
command to use `cargo llvm-cov` to generate a coverage report from the
raw profiles.
The workflow is:
```
./mach build --coverage [--profile <cargo_profile>]
# Or test-wpt or test-devtools
./mach run --coverage [--profile <cargo_profile>]
# Note, that coverage-report needs to know the cargo build profile.
./mach coverage-report [--profile <cargo_profile>] [optional parameters for cargo-llvm-cov]
```
According to the LLVM documentation on source based coverage, the
optimization profile should not influence the accuracy of the coverage
profile, so we can gather coverage data from optimized builds.
Note that `./mach test-devtools --coverage` will not produce any
coverage profiles yet, since the test runner kills the servo binary,
which prevents writing the profile data at shutdown.
The same problem also affects `test-wpt` with `servodriver`, which will
be fixed by https://github.com/servo/servo/pull/40455.
Testing: Manually tested. A CI workflow to test wpt coverage will be
added in a follow-up PR.
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
cargo places the shared library artifact under
`target/<target_triple>/<profile>/libservoshell.so`. The openharmony
prefix, is something we do in mach for the hap, and is not relevant for
the .so (essentially a copy-paste mistake).
Can be compared by looking at the job run of
https://github.com/servo/servo/actions/runs/19125758002/job/54655560361
which failed due to the wrong path.
Testing: Not tested
Fixes: #40467
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
In the review for #40369, the name of a command-line argument was
changed, but not the code that accessed the argument. This meant that
the unexpected log was never dumped, even when there are stable
unexpected results. This change fixes that.
Testing: This should ensure that the `stable-unexpected-results-linux`
artifact
is not empty when running WPT on CI.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Comprehensively sets the version across Cargo.toml and all
installer/config files (i.e. `Info.plist`, `Servo.wxs.mako`, etc.)
Testing: Manual
Fixes: #40312
---------
Signed-off-by: Ashwin Naren <arihant2math@gmail.com>
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Co-authored-by: Jonathan Schwender <schwenderjonathan@gmail.com>
This change switches the default test runner for WPT to be WebDriver,
enabling testdriver tests by default. In addition, it update results to
reflect the ones that you would expect when running with WebDriver.
While there are some failures that require more investigation, in
general the differences in results are fairly explicable.
Testing: This change modifies the way that tests are run and is thus
tested by all WPT-like tests.
Fixes: #34683
---------
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This change ensures that the filtered raw log file of WPT results only
contains stable unexpected results and uploads a resulting aggregated
log file as an artifact. This artifact can be used to generate new
baselines of *only* stable unexpected results, so is useful for updating
results given a CI run. A future change will add a `mach` command to do
this automatically.
Testing: The CI should run the code to produce this artifact for every
WPT run.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
- Compile the nightly in production mode
- Additionally also upload `libservoshell.so` as an artifact. This makes
it easier for interested users on HarmonyOS to test the demo app, since
we can't provide a signed version for HarmonyOS, but users can self-sign
easily if we provide the pre-compiled binary.
Testing: Not tested
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Instead of ignoring the `--headless` argument to the WPT test runner,
properly use it to turno on headless mode in Servo when its provided.
Additionally, when more than a single test is run, turn on headless mode
automatically to preserve existing behavior.
Although this change is for the legacy test driver, this will allow the
WebDriver test runner to run properly without providing the `--headless`
argument.
Testing: This modifies the way the test harness works, but the test
harness is untested.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
The patch implements Compression (https://compression.spec.whatwg.org/)
with the compression and decompression provided by the `flate2` crate
(https://crates.io/crates/flate2).
`flate2` supports several different backends, controlled through the
crate's features. By default, it uses `miniz_oxide`
(https://crates.io/crates/miniz_oxide).
`flate2` provides three modules `read`, `write` and `bufread` which work
on instances of the `std::io::Read`, `std::io::Write` and
`std::io::Bufread` traits, respectively. The `write` module is chosen in
the patch since it matches the streaming model in the specification.
Testing: Enable WPT for Compression API, and introduce WPT expectation.
---------
Signed-off-by: Kingsley Yung <kingsley@kkoyung.dev>
On recent versions of HarmonyOS the previous JIT detection does not
appear to work anymore, since the second `mmap` succeeds. This moves the
failure in spidermonkey to a later point, when SM remaps memory to be
writable. On recent versions of HOS I observed that the `PROT_WRITE`
permission was silently ignored by mprotect, which lead to a crash later
on the first attempt to write to the memory region.
It's not exactly easy to determine if one can write to a memory location
(if mprotect lies to you), but we can use `read` to check if the memory
region is writable (without triggering a segfault), since `read` will
return an error for invalid addresses (which is possible since the
writing is handled in the kernel).
Since this solution does add more unsafe code than before, we only use
this detection on OpenHarmony, although we could use it on more
platforms later if there are other platforms which also may have JIT
forbidden.
Testing: Removing the explicit `--pref js_disable_jit=true` means the
detection is tested in CI.
Fixes: #40029
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Signed-off-by: Jonathan Schwender <55576758+jschwe@users.noreply.github.com>
Co-authored-by: Sam <16504129+sagudev@users.noreply.github.com>
Use Release mode for gradle for production builds and Debug for any
other builds.
This fixes an exception when building for android with a custom cargo
profile.
`SERVO_TARGET_DIR` is read in `apk/jni/Android.mk` and specifies the
folder
libservoshell.so is expected to be in.
Testing: Tested manually with `./mach build --android --production`
Fixes: #34564
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Signed-off-by: Jonathan Schwender <55576758+jschwe@users.noreply.github.com>
Co-authored-by: Mukilan Thiyagarajan <mukilanthiagarajan@gmail.com>
Changes the asset name on github releases to include the platform. We
map the default platform name (e.g. linux) to something closer to the
target triple (x86_64-linux-gnu), since the architecture and env are
important details.
This allows uploading more than one artifact per OS, which is important
for macos nightlies now.
We additionally continue uploading the artifacts under the old name,
which effectively duplicates assets, to keep the asset name stable
during a transition period. This is useful for easily bisecting servo
using release assets, without needing to consider name changes.
Testing: Tested by manually triggering a release workflow
Fixes: #39973
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Co-authored-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Only do x86 builds on macos on push and on nightly builds. Also test
building libservo on arm mac instead of x86 mac. Also fixes an issue
when merging macos arm jobs in the try parser.
The main goal of this PR is to improve CI merge times again (since the
concurrency limit of github-hosted runners, has been slowing down the MQ
a lot).
Testing: This changes a CI workflow.
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Add --skip-nextest parameter to bootstrap and apply it where nextest is
not required.
Testing: Not tested.
Fixes: #39942
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Removes files that once were used for testing tidy checks which are no
longer performed, such as:
- json formatting
- lock files check which was superseded by cargo deny,
- checking for long lines and multiline strings
- sorting alphabetically #![feature(...)] statements
Testing: Not required, unused files removed
Signed-off-by: Gae24 <96017547+Gae24@users.noreply.github.com>
Nextest is a powerful test runner, with many advantages over cargo test.
Among others it will run each test in a separate process, provide a
summary of the completed test execution, supports output formats like
JUnit, and can handle flaky test by retrying. This PR does not use most
of these advanced features yet though, that will be left to future PRs.
The change also uncovered racy tests in `net` which rely on someone
initializing the `ASYNC_RUNTIME` before the test in questions is run. By
explicitly accessing the lazy static in a common setup routine, we make
sure it is initialized.
CLI differences of `cargo test` vs `cargo nextest`:
- The `--profile` option in nextest is not the cargo profile, but refers
to nextest configuration profiles.
- `--nocapture` is a direct argument to nextest, not to the testharness,
so we can remove the preceding `--` seperator.
Testing: Tested by running unit-tests in CI
Fixes: #39797
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Add a CI workflow to determine the code coverage. For now we only run
unit-tests, wpt-tests with code coverage can be added later.
We use `lld` to avoid OOM errors during linking (with `ld`, we would
need to reduce the build-concurrency drastically to prevent OOM errors
in CI)
For now, coverage would run during `full` and on-demand (i.e explicitly
requested via try). This workflow failing should not influence the
merge-queue, since we intentionally don't add this workflow to the
required status checks (yet).
Future work: It still needs some investigation to figure out if we can
get pull-request comments from the codecov bot, for coverage runs on try
branches.
Testing: [try
run](https://github.com/servo/servo/actions/runs/18431480782/job/52519348479),
[codecov report from the try
run](f41a50f321)
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Signed-off-by: Jonathan Schwender <55576758+jschwe@users.noreply.github.com>
Co-authored-by: Sam <16504129+sagudev@users.noreply.github.com>
Add a `--code-coverage` option for `./mach test-unit` and an option to
pass additional arguments.
The additional options are required to e.g. pass `--codecov
--output-path=codecov.json` to generate a codecov compatible report in
CI.
We use [cargo-llvm-cov](https://github.com/taiki-e/cargo-llvm-cov) to
simplify our workflow, since it handles both the required compile flags
and creating the report.
Testing: Manual testing of the new flags
First step towards https://github.com/servo/servo/issues/37768
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
This implements LazyDOMString (from now on DOMString) as outlined in
https://github.com/servo/servo/issues/39479.
Constructing from a *mut JSString we keep the in a
RootedTraceableBox<Heap<*mut JSString>> and transform
the string into a rust string if necessary via the `make_rust_string`
method.
Methods used in script are implemented on this string. Currently we
transform the string at all times.
But in the future more efficient implementations are possible.
We implement the safety critical sections in a separate module
DOMStringInner which allows simple constructors, `make_rust_string` and
the `bytes` method.
This method returns the new type `EncodedBytes` which contains the
reference to the underlying string in either format.
Testing: WPT tests still seem to work, so this should test this
functionality.
---------
Signed-off-by: Narfinger <Narfinger@users.noreply.github.com>
We were capturing test output (such as stack traces) by using the "test"
key in the data provided to the `process_output` log handler. This
doesn't work for WebDriver mode because the test runner runs in another
process entirely and it may reuse Servo executable instances for tests.
This change maps browser process output to tests using the "browser_pid"
key that is often provided while in WebDriver mode.
Testing: This modifies the test harness, so the tests are running tests.
Fixes: #39176
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Check for the pid of servo for test-speedometer-ohos to see if it did
not crash and abort early if it crashed.
Testing: Ran test-speedometer-ohos with closing servo and without
closing servo on the phone.
Fixes: https://github.com/servo/servo/issues/37742
Signed-off-by: Narfinger <Narfinger@users.noreply.github.com>
Format the package hash files to include the package hash and filename
separated by two spaces, ensuring compatibility with the `sha256sum -c`
and `shasum -c` command.
---
Before:
```
╰─❯ sha256sum -c servo-latest.tar.gz.sha256
sha256sum: servo-latest.tar.gz.sha256: no properly formatted checksum lines found
```
After:
```
╰─❯ sha256sum -c servo-latest.tar.gz.sha256
servo-latest.tar.gz: OK
```
Signed-off-by: Integral <integral@member.fsf.org>
Content tests have been replaced by web platform tests 10 years ago (see
9be71b941f).
I think it is fair to assume that no one is going to run the relevant
subcommand anymore. Let's remove it.
Signed-off-by: Simon Wülker <simon.wuelker@arcor.de>
one of the flaky failure modes we found in #38658 was that on linux,
geckordp occasionally fails to connect to servoshell’s devtools server.
this happens despite our preliminary connect check passing, which should
imply that the devtools server is listening and ready to use. we closed
the issue without any fix for that failure mode, because we were
ultimately unable to reproduce it, but it still happens in the wild
(#39273). we’ve now found a way to reproduce it, and we think it’s
caused by a race that occurs when moving from one test to the next. for
example:
- test 1 finishes
- we send SIGTERM to test 1’s servoshell, but it does not stop its
devtools server yet
- test 2 begins
- we spawn test 2’s servoshell, but it does not start its devtools
server yet
- we try to do our preliminary connects, and it succeeds against test
1’s servoshell immediately (the failure logs on GitHub never make this
clear, due to some kind of buffering problem that delays the `.` and `+`
outputs)
- test 1’s servoshell stops its devtools server
- we try to do our actual connect, and it fails because no devtools
server is listening
- test 2 fails
very rarely, one test’s servoshell may even fail to start its devtools
server, which we think happens because the previous test’s servoshell is
still listening. this has only ever happened once, and we’ve been unable
to reproduce it since, but we think it’s caused by the same kind of
race. for example:
- test 1 finishes
- we send SIGTERM to test 1’s servoshell, but it does not stop its
devtools server yet
- test 2 begins
- we spawn test 2’s servoshell, but it does not start its devtools
server yet
- test 2’s servoshell tries to start its devtools server, but fails
because test 1’s servoshell is still listening
- test 2 fails
in both cases, the failure can be explained by the fact that we send
SIGTERM to the previous test’s servoshell without actually waiting for
the process to exit. this patch ensures that we wait, and also moves all
of the output we do in the test suite from stdout to stderr to avoid it
getting mangled in GitHub Actions.
Testing: see [this
comment](https://github.com/servo/servo/pull/39309#issuecomment-3291007931)
(before) vs [this
comment](https://github.com/servo/servo/pull/39309#issuecomment-3291188997)
(after)
Fixes: #39273
Signed-off-by: Delan Azabani <dazabani@igalia.com>
Architecture:
```
Desktop (Test Controller) OHOS Device (Test Target)
┌─────────────────────────┐ ┌─────────────────────────┐
│ WPT Server (port 8000) │ │ Servo Browser │
│ Test Runner Script │---->│ WebDriver Server (7000) │
│ HDC Port Forwarding │ │ Test Execution │
└─────────────────────────┘ └─────────────────────────┘
```
After the test is finished, the script will parse the results and print
them in a readable format.
Tried to handle as many errors as possible and find workarounds for each
error to ensure the testing can be completed, or at least provide
comprehensive logs or information to identify exactly where the problem
is. Note that the used ports are just for testing; you can change them
to any other available ports, but make sure that the ports are
consistent in the script and given commands.
To run a WPT test on an OHOS device, you need to:
1. Connect OHOS device to the desktop via a cable (didn't try any other
way of communication)
2. Build and deploy servo with the changes in this PR using
[servoDemo](https://github.com/jschwe/ServoDemo). You can find there the
instructions to build and deploy servo to OHOS device.
3. While deploying servo to OHOS you need to ensure WebDriver is enabled
with the argument --webdriver=7000
4. Ensure OHOS SDK with HDC in PATH
5. Start WPT server on the desktop on a different terminal in servo
directory: ```bash python -m wpt serve --port 8000 ```
6. Update desktop IP in test scripts:
```python
desktop_ip = "192.168.1.100" # Your desktop's IP
```
You can find your desktop IP with:
```bash
# Windows
ipconfig | findstr "IPv4"
# macOS/Linux
ifconfig | grep "inet "
```
script can be modified to detect the desktop's IP automatically ...
later.
7. Run tests using the new mach command:
```bash
./mach test-ohos-wpt \
--test <test relative path> \
--webdriver-port 7000 \
--wpt-server-port 8000 \
--verbose
```
The script will:
1. Set up HDC port forwarding and reverse port forwarding for WPT
automatically
2. Connect to WebDriver server on the device
3. Navigate to the test URL
4. Wait for test completion
5. Show test results
Troubleshooting common Issues and Solutions:
1. HDC command not found:
- Install OHOS SDK and add HDC to PATH
- Verify: `hdc --version`
2. Failed to connect to WebDriver:
- Ensure Servo is running with `--webdriver=7000` argument
- Check device connection: `hdc list targets`
- Verify port forwarding: `hdc fport ls`
- Test WebDriver directly: `curl http://localhost:7000/status`
3. Failed to navigate to test URL:
- Update `desktop_ip` in the script
- Ensure both devices are on same network or connected via cable
- Test connectivity: ping from device to desktop
4. Test timeouts:
- Increase timeout in script (default: 30 seconds)
- Check if test requires specific dependencies
- Verify WPT server is serving the test file
---------
Signed-off-by: abdelrahman1234567 <abdelrahman.hossameldin.awadalla@huawei.com>
Signed-off-by: Euclid Ye <yezhizhenjiakang@gmail.com>
Co-authored-by: Euclid Ye <yezhizhenjiakang@gmail.com>
Since WebDriver is quite stable now, enable the `webdriver/classic`
tests on wpt CI.
---------
Signed-off-by: PotatoCP <Kenzie.Raditya.Tirtarahardja@huawei.com>
In the sources list tests, we assert that the sources for each target
are given to us in the same order as we specified in the test case, but
this is only true for classic <script> and <script src>. ES module
scripts and async/defer scripts are loaded asynchronously, so we can’t
rely on the order being the same every time.
this patch changes the test assertions to use a frozen multiset for each
target’s sources, rather than a frozen list (tuple), so the sources can
appear in any order but must still appear the expected number of times.
we also change the test assertions to use a multiset
([Counter](https://docs.python.org/3/library/collections.html#counter-objects))
of frozen multisets, rather than a set of multisets, so now two targets
can have the same set of sources without breaking tests.
Testing: this patch improves existing tests, but does not change
coverage
Fixes: part of #38658
---------
Signed-off-by: atbrakhi <atbrakhi@igalia.com>
Co-authored-by: delan azabani <dazabani@igalia.com>
In #38933 we removed `start_web_server` but
`test_source_breakable_lines_and_positions_with_functions` was not
updated because it was added later. In this patch we remove
`start_web_server` in that test as well.
Testing: fixes an existing test
Fixes: Part of #36325
Signed-off-by: atbrakhi <atbrakhi@igalia.com>
Co-authored-by: delan azabani <dazabani@igalia.com>
#38614 was reverted due to CI flakiness, but it also included several
improvements to the devtools tests. this patch relands those
improvements, described below.
we make three changes that speed up the devtools tests from 73 → 65 → 56
→ 51 seconds:
- we replace the hardcoded sleep(1) after starting servoshell with a
loop that waits until the devtools port is open (this also fixes
intermittent failures when servoshell starts too slowly, especially on
macOS)
- we start the internal web servers once, and reuse them across all
tests
- we run servoshell in headless mode (this is also required because most
CI runners have no GUI)
and we fix two bugs that cause very noisy but not very interesting error
messages:
- in the test code, we use a [context
manager](https://docs.python.org/3/reference/datamodel.html#context-managers)
to ensure the devtools client is disconnected unconditionally, even if
test methods or assert helper methods raise exceptions (this was causing
errors on all platforms)
- in the devtools server, we treat “connection reset” errors when
reading from the client like a normal EOF, rather than as a failure
(this was causing errors on Windows)
on self-hosted linux builds, there are still spurious error messages
like the following, but we can fix them later:
```
error: XDG_RUNTIME_DIR not set in the environment.
libEGL warning: egl: failed to create dri2 screen
```
Testing: this patch improves the devtools tests, but does not change
their coverage
Fixes: part of #36325
---------
Signed-off-by: Delan Azabani <dazabani@igalia.com>
Signed-off-by: atbrakhi <atbrakhi@igalia.com>
Co-authored-by: atbrakhi <atbrakhi@igalia.com>
Currently speedometer results are parsed by parsing the console output
from stdout (or a file in the case of ohos). Currently json decode
errors just cause mach to crash. Instead print an error message, point
to the problematic location and exit.
A crash can happen if something else also prints, e.g. on macos, we have
the `xx threads are still running` message at shutdown. Hence this PR
doesn't really fix the unreliable nature of the current implementation,
but at least adds a helpful error message, which would point people in
the right direction.
Testing: test-speedometer is run in CI
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
