Instead of attempting to manage the virtual environment ourselves, use
`uv` to manage the installation of dependencies.
Since we still have dependencies coming from upstream wpt, we use
`[tool.setuptool]` in our pyproject.toml to ensure that `uv` dynamically
installs our dependencies according to the requirements.txt files.
Additionally, this PR also reverts `--no-project` usage. `--no-project`
was added as a temporary workaround in
https://github.com/servo/servo/pull/37741.
It's not 100% clear to me what exactly the issue was, but
[apparently](https://github.com/servo/servo/pull/37741#pullrequestreview-2985666234)
the issue caused the build to break.
Removing the arg seems to work fine, except that we get a warning about
a missing `requiress-python` value in `pyproject.toml`.
Apparently it is good practice to specify the requirement as `>=` in th
pyroject, and lock the exact version via `uv pin` (which writes to
`.python_version`, where we already pin 3.11.
Testing: Should be covered by existing tests, which compile code on all
platforms.
---------
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Implements #40502
Adds `--yes` flag to mach bootstrap command which confirms installation
of all required packages.
Unlike `--force` flag (which also does that) it does NOT reinstall all
packages but only confirms them (if required at all).
Test on Ubuntu 24.04.2 LTS by removing safe development packages like
`libgstrtspserver-1.0-dev libges-1.0-dev gstreamer1.0-tools` and run
`mach bootstrap` twice to observe if `--yes` flags automatically install
all dependencies.
With this flag all dependencies were installed with any interaction but
removing some dependencies and running `./mach bootstrap` prompted me to
confirm `apt-get install` action.
---------
Signed-off-by: MCozhusheck <mackozuszek@gmail.com>
On NixOS, dynamically linked binaries from the Python venv (installed
via uv/pip) cannot run directly because they expect standard Linux
library paths that don't exist on NixOS.
This commit adds FHS wrappers that allow these binaries to run in an
FHS-compatible environment at runtime, without patching them.
I considered using the nixpkgs versions of ruff and pyrefly directly, or
overriding their derivations to match the versions in requirements.txt.
However, decided against it because:
- Version mismatches between nixpkgs and requirements.txt caused type
checking incompatibilities (pyrefly 0.34.0 vs 0.23.1)
- Building these tools from source in nix is slow and adds significant
time to nix-shell initialization (both are rust packages that take quite
some time to build)
Testing: just improvements to the NixOS development environment, no test
needed
**Before:**
```
➜ servo (main) ✔ nix-shell
➜ servo (main) ✔ ./mach fmt
Could not start dynamically linked executable: /home/dyego/coding/random/servo/.venv/bin/ruff
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld
➜ servo (main) ✔ ./mach test-tidy
➤ Checking config file (./servo-tidy.toml)...
➤ Checking directories for correct file extensions...
Could not start dynamically linked executable: ruff
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld
Error running mach:
['test-tidy']
The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.
If filing a bug, please include the full output of mach, including this error
message.
The details of the failure are as follows:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
File "/home/dyego/coding/random/servo/python/servo/testing_commands.py", line 322, in test_tidy
tidy_failed = tidy.scan(not all_files, not no_progress, github_annotations)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dyego/coding/random/servo/python/tidy/tidy.py", line 919, in scan
for error in errors:
File "/home/dyego/coding/random/servo/python/tidy/tidy.py", line 401, in check_ruff_lints
for error in json.loads(e.output):
^^^^^^^^^^^^^^^^^^^^
File "/nix/store/2g9b898aq9kmizmhmhbdip5mixrc5wrk-python3-3.11.14/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/2g9b898aq9kmizmhmhbdip5mixrc5wrk-python3-3.11.14/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/2g9b898aq9kmizmhmhbdip5mixrc5wrk-python3-3.11.14/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
➜ servo (main) ✔
```
(note that the `JSONDecodeError` is because we're trying to parse the
"NixOS cannot run dynamically linked executables intended..." string as
JSON)
**Now:**
```
➜ servo (fix-nix-mach) ✔ nix-shell
➜ servo (fix-nix-mach) ✔ ./mach fmt
➜ servo (fix-nix-mach) ✔ ./mach test-tidy
➤ Checking config file (./servo-tidy.toml)...
➤ Checking directories for correct file extensions...
➤ Checking type annotations in python files ...
➤ Skipping WPT lint checks, because no relevant files changed.
➤ Running `cargo-deny` checks...
➤ Checking formatting of Rust files...
➤ Checking formatting of python files...
➤ Checking formatting of toml files...
✅ test-tidy reported no errors.
➜ servo (fix-nix-mach) ✔
```
Signed-off-by: Dyego Aurélio <dyegoaurelio@gmail.com>
Add --skip-nextest parameter to bootstrap and apply it where nextest is
not required.
Testing: Not tested.
Fixes: #39942
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
This changes will introduce [flake8-annotations
(ANN)](https://docs.astral.sh/ruff/rules/#flake8-annotations-ann) for
python type annotation, this will make all thing related to function
strictly typed in python
This rule will start to affected this directory from now:
- /python -> Root directory
- /python/tidy
- /python/wpt
Testing: `./mach test-tidy`
Fixes: Not related to any issues
---------
Signed-off-by: Jerens Lensun <jerensslensun@gmail.com>
The new images published in servo/ci-runners#12 should have `uv`
installed already and the initial build of servo triggered during the
base image construction will force the installation of the Python
version mentioned at the time of the image construction (3.12). When
.python-version changes, we can no longer use the .venv baked into the
image and must recreate the it to avoid activating the environment.
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
This allows us to use `uv` for:
1. Installing a pinned Python version
2. Installing the dependency packages using `uv`'s pip compatible interface.
4. Bootstrapping `mach` without a Python installion on the host, using `uv
run`
This change also introduces a new 'composite' GitHub action to setup
python in the different CI workflows. There is no support for externally
managed python installations and virtual environments. These could be
added in the future.
Fixes#34095, #34547
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
This patch switches servo to use `uv` for both installing a pinned
Python version as well as installing the dependency packages using
`uv`'s pip compatible interface. It also introduces a new 'composite'
GitHub action to setup python in the different CI workflows.
There is no support for externally managed python installations and
virtual environments. These could be added in the future.
Fixes#34095
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
The default user name in Windows installations is of the form "FirstName
LastName", so it seems likely that there will be spaces in the user's
path. Based on my testing on Windows 11, the only Servo's bootstrap
script has trouble dealing with spaces in paths. This patch fixes that
by quoting such paths correctly. Our direct and indirect dependencies
seem to handle these without issue and Servo does build and run
correctly with this patch.
In this patch, the logic for gstreamer bootstrap now uses powershell
instead of directly invoking msiexec.exe via cmd.exe as I was unable to
get the installer to run correctly, even with quoting. Some extra hacks
were necessary to propagate the exit code correctly to mach.
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
I added this print statement, but after using `mach` for a while with
it, I think it is far too chatty. This change just removes the line.
Information is only printed when dependencies are installed.
We can now use the "new" pip resolver which should prevent the
installation of conflicting packages. Also, take this opportunity to
make bootstrap more resilient. Hash all dependencies to detect
situations where a newer marker file has been installed, but for an
older branch. This should ensure that dependencies are up to date even
when switching back and forth between older and new branches.
This also updates some dependencies to be the same as the ones used for
WPT tests, which is an issue caught be the resolver.
Fixes#10611.
It's often the case (especially with the taplo installation and on
Windows) that bootstrap is doing lots of stuff in the background for a
long amount of time. Without output it's hard to tell what exactly is
going on. This change adds more output to this process as well as
removing some Pythong 2.x era code.
1. Make the tidy output easier to follow
2. Integrate the WPT manifest cleanliness step into tidy
itself and don't run it if nothing has changed in the WPT
directory.
3. Fix an issue where Python test requirements were not installed,
which could cause issues with some modules not being found.
Fixes#30002.
* Replace virtualenv with Python's built-in venv.
* Apply Delan's suggestions and make a couple small fixes
- Fix a tidy warning about directories that don't exist
- Use shutil instead of the redundant get_exec_path
- Miscellaneous cleanups
* Fix typo in environment variable
* fix bug where pip still tries to the wrong site-packages
---------
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Co-authored-by: Delan Azabani <dazabani@igalia.com>
The version of `mozdebug` installable via pip is two years old and
Mozilla is slow to release new versions. It also uses `distutils` which
doesn't work on newer Python versions. Vendor it and stop using
`distutils` there.
* mach: use `importlib` module instead of `imp`
`imp` module has been deprecated since python 3.4
and has been removed in 3.12. The recommended alternative
is to use the `importlib` module that was introduced in
python 3.1
This is required to fix the CI failures in macos builds
since GitHub runner images for macos-13 now use python 3.12
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* mach: use `importlib` module instead of `imp`
`imp` module has been deprecated since python 3.4
and has been removed in 3.12. The recommended alternative
is to use the `importlib` module that was introduced in
python 3.1
This is required to fix the CI failures in macos builds
since GitHub runner images for macos-13 now use python 3.12
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
---------
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Servo is no longer completely vendored into Gecko. Instead parts of
Gecko are vendored into Servo. This change removes Python mach bootstrap
code that was written to accommodate the previous situation. It's no
longer necessary.
It seems that servo-tidy is only used by webrender in my GitHub
searches. WebRender could simply use `rustfmt` and the tidy on pypi
hasn't been updated since 2018. Converting tidy to a normal Python
package removes the maintenance burden of continually fixing the easy
install configuration.
Fixes#29094.
Fixes#29334.
On Windows with multiple Pythons installed, this was causing python2.7
to bootstrap a 3.7 virtualenv that it couldn't make use of.
PIP_NAMES wasn't used at all, and VIRTUALENV_NAMES ends up being unused
now.
The sys.platform check in need_pip_upgrade was doing effectively the same pip invocation regardless of what
platform we were evaluating to.
Additionally, removed some duplicate definitions of the python variable which was already in scope
higher up in the function.
