mirror of
https://github.com/servo/servo
synced 2026-04-25 17:15:48 +02:00
Updated Mozlando Oxidation (markdown)
Lars Bergstrom
@@ -4,286 +4,146 @@ Thursday, 12/10, 11:30-12:30
|
||||
Europe 2
|
||||
|
||||
# Agenda
|
||||
|
||||
Individual project status
|
||||
|
||||
MooV parser
|
||||
|
||||
is in-tree, linux 64, inboud on Mac. Next are Windows and Android
|
||||
|
||||
Help on Windows, Android and B2G would be good! Bottleneck is CI.
|
||||
|
||||
230k executions in telemetry
|
||||
|
||||
rust-url
|
||||
|
||||
necko tests pass
|
||||
|
||||
IDNA support currently being worked on (PR open)
|
||||
|
||||
percent decoding of the hostname is quite different - until recently we didn't do it at all for hostnames, and now we only decode valid hostname characters (bug 412457)
|
||||
|
||||
needed to make rust-url accept other schemes, such as resource:// in order to be able to run the browser, since some schemes also go through the parser (PR pending)
|
||||
|
||||
Have not yet done a full 'try' run, which should flush remaining issues
|
||||
|
||||
Did the build using cargo; still waiting for the rest of the build system integration
|
||||
|
||||
What about BMP decoder? (bholley)
|
||||
|
||||
nnethercote just rewrote in C++, but at least with a big test suite
|
||||
|
||||
Isn't the MP4 parser bad?
|
||||
|
||||
Have both of them in-tree, run both for now. So still fixing stagefright security problems.
|
||||
|
||||
Platform support status - where are we at now? What blockers remain?
|
||||
|
||||
Rust 1.5 stable just hit!
|
||||
|
||||
Gives us multi-file crates,
|
||||
|
||||
Dependency generation for incremental builds.
|
||||
|
||||
jemalloc (fine to redirect)
|
||||
|
||||
Can now redirect the Rust code to use the standard symbols in Gecko. nfroyd!
|
||||
|
||||
work remaining for try builds
|
||||
|
||||
rpath/origin support to avoid LD_LIBRARY_PATH in try
|
||||
|
||||
Official rust binaries don't set LD_LIBRARY_PATH; we build a custom snapshot for the try servers
|
||||
|
||||
status for each tier 1 platform
|
||||
|
||||
Rust supports all tier 1 platforms; just need our build system integrations in space there (including OSX10.6, Linux64; Windows, Android, B2G need help. Just have to add more libstds)
|
||||
|
||||
For some platforms like Android that just involves adding new targets to the build system and putting the additional libraries in place. Any volunteers? nfroyd!
|
||||
|
||||
How much work would it take to make just the style system build on stable Rust (not require unstable nightly features)?
|
||||
|
||||
pcwalton: Ms2ger has done work to move us off of non-necessary unstable features. Do you know how much unstable stuff layout is using?
|
||||
|
||||
Ms2ger: Most things in the layout crate could be removed.
|
||||
|
||||
Manishearth: I did an audit (https://gist.github.com/Manishearth/967bc78db5c751c38d03#layout, many of these have stabilized since) a few months ago. Most of the unstable features are left over from pre-1.0 code and we just haven't bothered changing them. Ignoring plugins, it's mostly easy. Plugins could be replaced with codegen.
|
||||
|
||||
pcwalton: Do we even use any plugins in layout?
|
||||
|
||||
(discussion to continue offline)
|
||||
|
||||
Code / optimize / debug workflow
|
||||
|
||||
How do we handle PGO build interactions (if in LLVM, can be added)
|
||||
|
||||
We do PGO on Win (using MSVC) and Linux (GCC).
|
||||
|
||||
I don't know what it means to do this. We use LLVM codegen across the board. As long as you treat us like other native libraries it should work out.
|
||||
|
||||
The Rust code may not get the full benefit of PGO but it shouldn't cause problems. We currently compile some third-party code like ICU without PGO.
|
||||
|
||||
LLVM has PGO support but we haven't done the legwork to use it.
|
||||
|
||||
PGO is useful for devirtualization which isn't as important in Rust.
|
||||
|
||||
It can also change optimizations based on hot vs. cold paths.
|
||||
|
||||
bz: Once we have a style system in Rust, having PGO for the style system *would* be nice.
|
||||
|
||||
How well does Rust work with the Gecko gdb/lldb/profiling (native and custom)?
|
||||
|
||||
Whatever LLVM supports, we support. I haven't seen any problems with standard tools.
|
||||
|
||||
rr works with Servo.
|
||||
|
||||
What about Windows?
|
||||
|
||||
A lot of the Windows stuff is based on LLVM, whose PDB support is relatively immature. We're keeping up to date with LLVM to benefit from new improvements.
|
||||
|
||||
Currently can't inspect local variables; sometimes stack traces are weird.
|
||||
|
||||
vlad: Maybe use the new stuff?
|
||||
|
||||
That uses the Clang front-end but uses MSVC++ for codegen; we can't use that for Rust.
|
||||
|
||||
There are caveats. We emit DWARF and masquerade as C++. The fidelity is low. There are person-years of work to get full mature support.
|
||||
|
||||
But it's good enough in practice for e.g. debugging Servo.
|
||||
|
||||
We should validate all this; not just debugging but also profiling, crash reporting.
|
||||
|
||||
If there are specific areas that need fixing, ask management for resources to do it.
|
||||
|
||||
Rust team's priority right now is developer ergonomics and tooling.
|
||||
|
||||
Diagnosing bugs reported "in the wild"
|
||||
|
||||
Crash reporting (how do stack traces look; are symbols generated/stored properly)
|
||||
|
||||
We got a reasonable stack trace from a crash report in a test run.
|
||||
|
||||
How do we test the full workflow of crash reporter -> crash-stats.mozilla.com?
|
||||
|
||||
bz: Introduce a crash behind a pref in nightly :-)
|
||||
|
||||
How do we make sure bugs are fixed in both engines?
|
||||
|
||||
upstream repo for create; changes land there first?
|
||||
|
||||
Once we have real users, we have less flexibility to change things without regard for impact on them.
|
||||
|
||||
This impacts code like rust-url or Servo layout code that's shared between Gecko and Servo (or other Rust projects)
|
||||
|
||||
We have this problem with a lot of third-party code used in mozilla-central.
|
||||
|
||||
We vendor everything. It makes our life easier in some ways, in others harder.
|
||||
|
||||
Servo uses cargo rather than vendoring things.
|
||||
|
||||
Maybe we can have a "cargo vendor" tool or similar that integrates with cargo to sync in-tree copies with upstream repos.
|
||||
|
||||
Or a more general (not Rust-specific) tool for syncing/diffing m-c directories against their upstream sources.
|
||||
|
||||
Servo has a mach command to do two-way sync between our vendored web-platform-tests and upstream.
|
||||
|
||||
bholley: we need better tooling in general for getting fixes in deeply nested dependencies deployed to downstream code.
|
||||
|
||||
Instead of one repo per crate, we could have one repo with many crates, if that helps.
|
||||
|
||||
acrichto: Idiomatic Rust practice is the opposite of "vendor everything." It's much easier if you can find ways to work the Rust ecosystem way.
|
||||
|
||||
ted: I understand that but I don't think we're going to go all the way there. There may be somewhere halfway.
|
||||
|
||||
gps: People really want bi-directional multi-vcs multi-repository "syncing"
|
||||
|
||||
Rust, Servo, media, ICU, WebRTC, web-platform-tests, /devtools, /browser, ....
|
||||
|
||||
Quilt/MQ like local patches
|
||||
|
||||
gps: devs can pull from crates.io. builders absolutely cannot hit crates.io.
|
||||
|
||||
ms2ger: We want this for devs' local builds too so they don't need to be connected after downloading/cloning m-c.
|
||||
|
||||
m-c tree integration
|
||||
|
||||
Vendoring and updating crates.io/git dependencies
|
||||
|
||||
Are there other constraints you can pass along?
|
||||
|
||||
Build one huge crate that builds all the Gecko/Rust code into one world
|
||||
|
||||
Not sure on the compilation model; need to clean it up a test rust-url example set of patches
|
||||
|
||||
implicitly trust code review from Servo/Rust code / module policy concerns
|
||||
|
||||
Common to 3rd party code. Rubber-stamp review? e.g., libjpeg... rely on push to try?
|
||||
|
||||
Expect responsive upstream for servo code
|
||||
|
||||
bisection
|
||||
|
||||
simonsapin: Should we import commit history?
|
||||
|
||||
gps: There are bots on FFOS that do automated commits
|
||||
|
||||
It periodically pulls from upstream, and lands the new revision if it passes tests. So every few hours it's updated to the latest upstream
|
||||
|
||||
ted: chromium does this, too via trybots
|
||||
|
||||
gps: Can't just pull the full commit histroy because it might not be commit-level bisectable. Can roll upstream pushes into commits.
|
||||
|
||||
Both Servo and mozilla-central are "push-level" bisectable but not commit-level. We try, but no automation enforcing that intermediate commits build.
|
||||
|
||||
bz: Once we have Servo style code in Gecko we need to have process like "Intent to Ship"
|
||||
|
||||
* Individual project status
|
||||
* MooV parser
|
||||
* is in-tree, linux 64, inboud on Mac. Next are Windows and Android
|
||||
* Help on Windows, Android and B2G would be good! Bottleneck is CI.
|
||||
* 230k executions in telemetry
|
||||
* rust-url
|
||||
* necko tests pass
|
||||
* IDNA support currently being worked on (PR open)
|
||||
* percent decoding of the hostname is quite different - until recently we didn't do it at all for hostnames, and now we only decode valid hostname characters (bug 412457)
|
||||
* needed to make rust-url accept other schemes, such as resource:// in order to be able to run the browser, since some schemes also go through the parser (PR pending)
|
||||
* Have not yet done a full 'try' run, which should flush remaining issues
|
||||
* Did the build using cargo; still waiting for the rest of the build system integration
|
||||
* What about BMP decoder? (bholley)
|
||||
* nnethercote just rewrote in C++, but at least with a big test suite
|
||||
* Isn't the MP4 parser bad?
|
||||
* Have both of them in-tree, run both for now. So still fixing stagefright security problems.
|
||||
* Platform support status - where are we at now? What blockers remain?
|
||||
* Rust 1.5 stable just hit!
|
||||
* Gives us multi-file crates,
|
||||
* Dependency generation for incremental builds.
|
||||
* jemalloc (fine to redirect)
|
||||
* Can now redirect the Rust code to use the standard symbols in Gecko. nfroyd!
|
||||
* work remaining for try builds
|
||||
* rpath/origin support to avoid LD_LIBRARY_PATH in try
|
||||
* Official rust binaries don't set LD_LIBRARY_PATH; we build a custom snapshot for the try servers
|
||||
* status for each tier 1 platform
|
||||
* Rust supports all tier 1 platforms; just need our build system integrations in space there (including OSX10.6, Linux64; Windows, Android, B2G need help. Just have to add more libstds)
|
||||
* For some platforms like Android that just involves adding new targets to the build system and putting the additional libraries in place. Any volunteers? nfroyd!
|
||||
* How much work would it take to make just the style system build on stable Rust (not require unstable nightly features)?
|
||||
* pcwalton: Ms2ger has done work to move us off of non-necessary unstable features. Do you know how much unstable stuff layout is using?
|
||||
* Ms2ger: Most things in the layout crate could be removed.
|
||||
* Manishearth: I did an audit (https://gist.github.com/Manishearth/967bc78db5c751c38d03#layout, many of these have stabilized since) a few months ago. Most of the unstable features are left over from pre-1.0 code and we just haven't bothered changing them. Ignoring plugins, it's mostly easy. Plugins could be replaced with codegen.
|
||||
* pcwalton: Do we even use any plugins in layout?
|
||||
* (discussion to continue offline)
|
||||
* Code / optimize / debug workflow
|
||||
* How do we handle PGO build interactions (if in LLVM, can be added)
|
||||
* We do PGO on Win (using MSVC) and Linux (GCC).
|
||||
* I don't know what it means to do this. We use LLVM codegen across the board. As long as you treat us like other native libraries it should work out.
|
||||
* The Rust code may not get the full benefit of PGO but it shouldn't cause problems. We currently compile some third-party code like ICU without PGO.
|
||||
* LLVM has PGO support but we haven't done the legwork to use it.
|
||||
* PGO is useful for devirtualization which isn't as important in Rust.
|
||||
* It can also change optimizations based on hot vs. cold paths.
|
||||
* bz: Once we have a style system in Rust, having PGO for the style system *would* be nice.
|
||||
* How well does Rust work with the Gecko gdb/lldb/profiling (native and custom)?
|
||||
* Whatever LLVM supports, we support. I haven't seen any problems with standard tools.
|
||||
* rr works with Servo.
|
||||
* What about Windows?
|
||||
* A lot of the Windows stuff is based on LLVM, whose PDB support is relatively immature. We're keeping up to date with LLVM to benefit from new improvements.
|
||||
* Currently can't inspect local variables; sometimes stack traces are weird.
|
||||
* vlad: Maybe use the new stuff?
|
||||
* That uses the Clang front-end but uses MSVC++ for codegen; we can't use that for Rust.
|
||||
* There are caveats. We emit DWARF and masquerade as C++. The fidelity is low. There are person-years of work to get full mature support.
|
||||
* But it's good enough in practice for e.g. debugging Servo.
|
||||
* We should validate all this; not just debugging but also profiling, crash reporting.
|
||||
* If there are specific areas that need fixing, ask management for resources to do it.
|
||||
* Rust team's priority right now is developer ergonomics and tooling.
|
||||
* Diagnosing bugs reported "in the wild"
|
||||
* Crash reporting (how do stack traces look; are symbols generated/stored properly)
|
||||
* We got a reasonable stack trace from a crash report in a test run.
|
||||
* How do we test the full workflow of crash reporter -> crash-stats.mozilla.com?
|
||||
* bz: Introduce a crash behind a pref in nightly :-)
|
||||
* How do we make sure bugs are fixed in both engines?
|
||||
* upstream repo for create; changes land there first?
|
||||
* Once we have real users, we have less flexibility to change things without regard for impact on them.
|
||||
* This impacts code like rust-url or Servo layout code that's shared between Gecko and Servo (or other Rust projects)
|
||||
* We have this problem with a lot of third-party code used in mozilla-central.
|
||||
* We vendor everything. It makes our life easier in some ways, in others harder.
|
||||
* Servo uses cargo rather than vendoring things.
|
||||
* Maybe we can have a "cargo vendor" tool or similar that integrates with cargo to sync in-tree copies with upstream repos.
|
||||
* Or a more general (not Rust-specific) tool for syncing/diffing m-c directories against their upstream sources.
|
||||
* Servo has a mach command to do two-way sync between our vendored web-platform-tests and upstream.
|
||||
* bholley: we need better tooling in general for getting fixes in deeply nested dependencies deployed to downstream code.
|
||||
* Instead of one repo per crate, we could have one repo with many crates, if that helps.
|
||||
* acrichto: Idiomatic Rust practice is the opposite of "vendor everything." It's much easier if you can find ways to work the Rust ecosystem way.
|
||||
* ted: I understand that but I don't think we're going to go all the way there. There may be somewhere halfway.
|
||||
* gps: People really want bi-directional multi-vcs multi-repository "syncing"
|
||||
* Rust, Servo, media, ICU, WebRTC, web-platform-tests, /devtools, /browser, ....
|
||||
* Quilt/MQ like local patches
|
||||
* gps: devs can pull from crates.io. builders absolutely cannot hit crates.io.
|
||||
* ms2ger: We want this for devs' local builds too so they don't need to be connected after downloading/cloning m-c.
|
||||
* m-c tree integration
|
||||
* Vendoring and updating crates.io/git dependencies
|
||||
* Are there other constraints you can pass along?
|
||||
* Build one huge crate that builds all the Gecko/Rust code into one world
|
||||
* Not sure on the compilation model; need to clean it up a test rust-url example set of patches
|
||||
* implicitly trust code review from Servo/Rust code / module policy concerns
|
||||
* Common to 3rd party code. Rubber-stamp review? e.g., libjpeg... rely on push to try?
|
||||
* Expect responsive upstream for servo code
|
||||
* bisection
|
||||
* simonsapin: Should we import commit history?
|
||||
* gps: There are bots on FFOS that do automated commits
|
||||
* It periodically pulls from upstream, and lands the new revision if it passes tests. So every few hours it's updated to the latest upstream
|
||||
* ted: chromium does this, too via trybots
|
||||
* gps: Can't just pull the full commit histroy because it might not be commit-level bisectable. Can roll upstream pushes into commits.
|
||||
* Both Servo and mozilla-central are "push-level" bisectable but not commit-level. We try, but no automation enforcing that intermediate commits build.
|
||||
* bz: Once we have Servo style code in Gecko we need to have process like "Intent to Ship"
|
||||
|
||||
Issues for the rust team
|
||||
|
||||
Rust has integer overflow checking in debug builds, disabled in optimized builds. Is there any static proof of no overflow?
|
||||
|
||||
No.
|
||||
|
||||
In media we'd like overflow checking enabled all the time.
|
||||
|
||||
Integer math in media is so fast compared to the overall video decoding process, we care much more about correctness than performance.
|
||||
|
||||
You can enable this in optimized builds with a compiler flag.
|
||||
|
||||
Would like to do it at the crate level
|
||||
|
||||
In Rust, even if you get an overflow, it's usually less serious than C++ because it can't cause buffer overrun/underrun or similar safety issues.
|
||||
|
||||
Array bounds checking is always enabled (in debug *and* release builds).
|
||||
|
||||
Some feedback that rustc's error messages are not good enough.
|
||||
|
||||
steveklabnik: Just recently the @RustLang twitter got two simultaneous messages saying "best error mesages ever!!!" and "terrible error messages! :("
|
||||
|
||||
We'd love specific feedback on any bad error messages. File issues. CC Niko and Manish
|
||||
|
||||
Compile times.
|
||||
|
||||
There are lots of tweaks you can do; we're always working on improving the compiler.
|
||||
|
||||
Biggest improvement will be incremental compilation. Lots of active work going on but it'll take a long time (several months) to be landed.
|
||||
|
||||
Our measurements show a steady improvement in compile speed over time, but there may be bad cases still.
|
||||
|
||||
If you have crates with especially bad compile time, send them to us to profile.
|
||||
|
||||
You can improve things by splitting into crates.
|
||||
|
||||
But then you don't get inlining across them (except with functions explicitly marked #[inline])
|
||||
|
||||
(Also, your functions are currently round-robined / otherwise arbitrarily divided, so what does get inlined (if it's not #[inline]) can be unpredictable.)
|
||||
|
||||
Servo team feels this pain a lot since we have some very very big crates that take minutes to compile, every time they change.
|
||||
|
||||
cargo and external build tools
|
||||
|
||||
incremental builds
|
||||
|
||||
expect results maybe by summer 2016
|
||||
|
||||
New rust toolchain every 6 weeks?
|
||||
|
||||
150MB. 400MB uncompressed.
|
||||
|
||||
Not good for places w/o reliable internet.
|
||||
|
||||
There's some discussion about recycling (C++) build artifacts for Gecko, so e.g. frontend developers can build JS changes without needing a C++ compiler.
|
||||
|
||||
Could leverage this for Rust too.
|
||||
|
||||
Of course you'd still need to build X megabytes of pre-built binaries.
|
||||
|
||||
At the beginning, we have little Rust code and frequent changes to the Rust toolchain, so faster to download the build artifacts.
|
||||
|
||||
Later, maybe we'll have more Rust code and more stable toolchain, so trade-off may change.
|
||||
|
||||
But we don't want to force upstream projects (e.g. Servo) to not track the latest Rust
|
||||
|
||||
At this point, benefit of upgrading is usually to get perf improvements. Feature-wise we could be happy with stable Rust.
|
||||
|
||||
Better to download Rust as source?
|
||||
|
||||
No, can add hours to initial build time.
|
||||
|
||||
We can just fedex a DVD with all the code every 6 weeks to contributors on slow networks.
|
||||
|
||||
How long can we keep Rust as an optional dependency?
|
||||
|
||||
Right now it's opt-in; eventually could be opt-out; at some point we'll have code that is Rust-only with no C++ fallback.
|
||||
|
||||
We don't want to ship two URL parsers forever (especially if one is untested and they are diverging)
|
||||
|
||||
This is a policy decision for the Build Config module owner
|
||||
|
||||
Could download partial updates for rustc
|
||||
|
||||
diff between releases currently likely to be quite large, possibly no improvement
|
||||
|
||||
* Rust has integer overflow checking in debug builds, disabled in optimized builds. Is there any static proof of no overflow?
|
||||
* No.
|
||||
* In media we'd like overflow checking enabled all the time.
|
||||
* Integer math in media is so fast compared to the overall video decoding process, we care much more about correctness than performance.
|
||||
* You can enable this in optimized builds with a compiler flag.
|
||||
* Would like to do it at the crate level
|
||||
* In Rust, even if you get an overflow, it's usually less serious than C++ because it can't cause buffer overrun/underrun or similar safety issues.
|
||||
* Array bounds checking is always enabled (in debug *and* release builds).
|
||||
* Some feedback that rustc's error messages are not good enough.
|
||||
* steveklabnik: Just recently the @RustLang twitter got two simultaneous messages saying "best error mesages ever!!!" and "terrible error messages! :("
|
||||
* We'd love specific feedback on any bad error messages. File issues. CC Niko and Manish
|
||||
* Compile times.
|
||||
* There are lots of tweaks you can do; we're always working on improving the compiler.
|
||||
* Biggest improvement will be incremental compilation. Lots of active work going on but it'll take a long time (several months) to be landed.
|
||||
* Our measurements show a steady improvement in compile speed over time, but there may be bad cases still.
|
||||
* If you have crates with especially bad compile time, send them to us to profile.
|
||||
* You can improve things by splitting into crates.
|
||||
* But then you don't get inlining across them (except with functions explicitly marked #[inline])
|
||||
* (Also, your functions are currently round-robined / otherwise arbitrarily divided, so what does get inlined (if it's not #[inline]) can be unpredictable.)
|
||||
* Servo team feels this pain a lot since we have some very very big crates that take minutes to compile, every time they change.
|
||||
* cargo and external build tools
|
||||
* incremental builds
|
||||
* expect results maybe by summer 2016
|
||||
* New rust toolchain every 6 weeks?
|
||||
* 150MB. 400MB uncompressed.
|
||||
* Not good for places w/o reliable internet.
|
||||
* There's some discussion about recycling (C++) build artifacts for Gecko, so e.g. frontend developers can build JS changes without needing a C++ compiler.
|
||||
* Could leverage this for Rust too.
|
||||
* Of course you'd still need to build X megabytes of pre-built binaries.
|
||||
* At the beginning, we have little Rust code and frequent changes to the Rust toolchain, so faster to download the build artifacts.
|
||||
* Later, maybe we'll have more Rust code and more stable toolchain, so trade-off may change.
|
||||
* But we don't want to force upstream projects (e.g. Servo) to not track the latest Rust
|
||||
* At this point, benefit of upgrading is usually to get perf improvements. Feature-wise we could be happy with stable Rust.
|
||||
* Better to download Rust as source?
|
||||
* No, can add hours to initial build time.
|
||||
* We can just fedex a DVD with all the code every 6 weeks to contributors on slow networks.
|
||||
* How long can we keep Rust as an optional dependency?
|
||||
* Right now it's opt-in; eventually could be opt-out; at some point we'll have code that is Rust-only with no C++ fallback.
|
||||
* We don't want to ship two URL parsers forever (especially if one is untested and they are diverging)
|
||||
* This is a policy decision for the Build Config module owner
|
||||
* Could download partial updates for rustc
|
||||
* diff between releases currently likely to be quite large, possibly no improvement
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user