The current version seem to be broken due to API change in one of the transitive dependency (urllib3) causing nightly builds to fail during package upload step with error:
```
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (C:\a\servo\servo\python\_virtualenv3.7\Lib\site-packages\urllib3\util\ssl_.py)
```
Use notify-py to send notifications
- Use notify-py to send notifications, but use a custom notifier on
Linux since transient (ie non-sticky) notifications are not supported.
- Add an icon to the notification.
- Don't send a notification after doing `./mach check` because that
can trigger notifications when `rust-analyzer` is working.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they just change build notifications.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
- Use notify-py to send notifications, but use a custom notifier on
Linux since transient (ie non-sticky) notifications are not supported.
- Add an icon to the notification.
- Don't send a notification after doing `./mach check` because that
can trigger notifications when `rust-analyzer` is working.
This change extends the `mach upload-nightly` command to
publish the nightly builds for all platforms as GH Release
assets.
The GH releases are made on a separate repository so
that we can persist older nightly builds without having
to accumulate git tags for them.
Some design tradeoffs in this approach are:
1. To allow the 'latest' link from servo.org to remain stable,
the release assets are named 'servo-latest.{ext}' instead of
containing the release tag/date.
2. The release is created as draft and published atomically
when all platforms have been built successfully. This allows
us to link to the 'latest' alias from servo.org while
gauranteeing that it contains builds for all platforms.
The other option here would be to have code in servo.org UI
that uses GH API to find the most recent release with a
successful build for a given platform.
3. The tags in the nightly repo are all based on the same
commit that has no relation to servo code base.
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
This will allow results to be formatted by other parts of the code (such
as the intermittent filtering) code. Previously, formatting was handled
in ServoHandler, which was a bit strange as it's really only necessary
for GroupingFormatter and the intermittent filtering code. This also
allows the results to be properly typed by the Python typing system.
https://github.com/servo/saltfs/pull/687 added support
for specifying environment variables in `buildbot_steps.yml`.
Update the servo-tidy buildbot_steps.yml linter to reflect this.
Use the voluptuous Python library (BSD 3-clause license) for validation
in lieu of a much larger hand-written implementation.
Update the tidy self tests to take into account the new error messages.
Now that MinGW and MSYS have been removed from the Windows builders,
bash is not available to run the previous upload_nightlies.sh script.
Convert the script to Python 2 for cross-platform support.
Additionally, switch to the `boto3` library for easy uploading
without needing to install `s3cmd`,
and move the code into mach for easy `boto3` installation
as the new `./mach upload-nightly` command.
Also, hard-code the paths to the packages instead of using
globs to look for them, as the paths are static.
(The paths used to contain timestamps,
but we now insert timestamps when uploading to S3
to improve reproducibility.)
The Salt bootstrapper invokes Salt during `./mach bootstrap`
to install Servo's build dependencies.
It uses salt-call pinned to the same version of Salt as used in saltfs.
Currently, the implementation uses gitfs and reads directly from
the master branch of the saltfs repo;
in the future this should be changed when the relevant Salt states
are moved in-tree as part of Dockerization for TaskCluster.
We have not Salted our Windows machines, so the existing Windows
bootstrappers are retained. Currently this is only tested on
Ubuntu Trusty.
Salt uses various system python libraries,
including `python-apt` on Debian-based OSes to interact with apt.
`python-apt` does not seem to be installable via a requirements.txt
file, and the versions available on PyPI are far behind the versions
installed on actual Ubuntu machines.
Additionally, adding `python-apt` as an unconditional python dependency
would add bloat for users of other OSes, and lead to more churn
as additional OSes are supported.
However, as `python-apt` is already installed via apt on these machines,
we can allow Salt to instead use the module by using
`--system-site-packages` for the python virtualenv.
We also add the `-I` flag to `pip install` to ensure we have a local,
untouched copy of any other python packages used.
However, because this prints system-level Python packages in scope,
it slightly breaks isolation, so it is important to always pin
all dependencies in the requirements files.
This commit adds tidy checks for buildbot_steps.yml, as well as unit
tests. These checks include:
* Checking buildbot_steps.yml can be parsed by a YAML loader
* buildbot_steps.yml does not contain duplicate keys
* buildbot_steps.yml keys map to a list of strings
When I originally rewrote Servo's mach bootstrapping (using virtualenv
w/ requirements.txt in #7103), I didn't specify mach as a requirement
because a new version hadn't been published in a while. Now that 0.6
is out, I asked the mach maintainers to publish a new version on PyPI,
so now we can fetch it like the other Python dependencies.
Fixes https://github.com/servo/servo/issues/10728.
- also update requriements "pyflakes" from 0.8 to 0.8.1 due to following issue
```
Traceback (most recent call last):
File "/Users/Askeing/software/servo/python/_virtualenv/bin/servo-tidy", line 5, in <module>
from pkg_resources import load_entry_point
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3084, in <module>
@_call_aside
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3070, in _call_aside
f(*args, **kwargs)
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3097, in
_initialize_master_working_set
working_set = WorkingSet._build_master()
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 653, in _build_master
return cls._build_from_requirements(__requires__)
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in
_build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 839, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pyflakes==0.8.1' distribution was not found and is required by servo-tidy
```
https://github.com/servo/servo/issues/861#issuecomment-209547756
"I think the most important concern is that it's possible to modify tidy.py
and see how those changes affect ./mach test-tidy with the fewest possible
intermediate steps." - jdm
This takes publishing complexity away from the contributor when testing
changes and makes it an infra problem instead, where it's much easier to
automate & saner to manage pypi credentials
Prior to this commit:
* Our Python dependency story was a bit of a mess. We had complete
Python packages (wheels and directories) living in-tree, despite
not having any changes from upstream. This is particularly bad because
`setup.py` never gets run on these packages which could (sometimes
silently) unintended breakage.
* Python virtual environments (virtualenv) were only utilized for
testing web-platform tests
After this commit:
* A single virtualenv (`python/_virtualenv`) is activated upon *every*
call to mach
* A requirements file (`python/requirements.txt`) is added to describe
the dependencies needed by Python modules in `python/`. The child
commit immediately following this will remove all the dependencies
no longer needed in-tree (for the sake of keeping this commit
readable).
Relevant to https://github.com/servo/servo/issues/861
Fixes https://github.com/servo/servo/issues/6999
