diff --git a/Mozlando-Oxidation.md b/Mozlando-Oxidation.md index c8b4993..ce922df 100644 --- a/Mozlando-Oxidation.md +++ b/Mozlando-Oxidation.md @@ -4,286 +4,146 @@ Thursday, 12/10, 11:30-12:30 Europe 2 # Agenda - - Individual project status - - MooV parser - - is in-tree, linux 64, inboud on Mac. Next are Windows and Android - - Help on Windows, Android and B2G would be good! Bottleneck is CI. - - 230k executions in telemetry - - rust-url - - necko tests pass - - IDNA support currently being worked on (PR open) - - percent decoding of the hostname is quite different - until recently we didn't do it at all for hostnames, and now we only decode valid hostname characters (bug 412457) - - needed to make rust-url accept other schemes, such as resource:// in order to be able to run the browser, since some schemes also go through the parser (PR pending) - - Have not yet done a full 'try' run, which should flush remaining issues - - Did the build using cargo; still waiting for the rest of the build system integration - - What about BMP decoder? (bholley) - - nnethercote just rewrote in C++, but at least with a big test suite - - Isn't the MP4 parser bad? - - Have both of them in-tree, run both for now. So still fixing stagefright security problems. - - Platform support status - where are we at now? What blockers remain? - - Rust 1.5 stable just hit! - - Gives us multi-file crates, - - Dependency generation for incremental builds. - - jemalloc (fine to redirect) - - Can now redirect the Rust code to use the standard symbols in Gecko. nfroyd! - - work remaining for try builds - - rpath/origin support to avoid LD_LIBRARY_PATH in try - - Official rust binaries don't set LD_LIBRARY_PATH; we build a custom snapshot for the try servers - - status for each tier 1 platform - - Rust supports all tier 1 platforms; just need our build system integrations in space there (including OSX10.6, Linux64; Windows, Android, B2G need help. Just have to add more libstds) - - For some platforms like Android that just involves adding new targets to the build system and putting the additional libraries in place. Any volunteers? nfroyd! - - How much work would it take to make just the style system build on stable Rust (not require unstable nightly features)? - - pcwalton: Ms2ger has done work to move us off of non-necessary unstable features. Do you know how much unstable stuff layout is using? - - Ms2ger: Most things in the layout crate could be removed. - - Manishearth: I did an audit (https://gist.github.com/Manishearth/967bc78db5c751c38d03#layout, many of these have stabilized since) a few months ago. Most of the unstable features are left over from pre-1.0 code and we just haven't bothered changing them. Ignoring plugins, it's mostly easy. Plugins could be replaced with codegen. - - pcwalton: Do we even use any plugins in layout? - - (discussion to continue offline) - - Code / optimize / debug workflow - - How do we handle PGO build interactions (if in LLVM, can be added) - - We do PGO on Win (using MSVC) and Linux (GCC). - - I don't know what it means to do this. We use LLVM codegen across the board. As long as you treat us like other native libraries it should work out. - - The Rust code may not get the full benefit of PGO but it shouldn't cause problems. We currently compile some third-party code like ICU without PGO. - - LLVM has PGO support but we haven't done the legwork to use it. - - PGO is useful for devirtualization which isn't as important in Rust. - - It can also change optimizations based on hot vs. cold paths. - - bz: Once we have a style system in Rust, having PGO for the style system *would* be nice. - - How well does Rust work with the Gecko gdb/lldb/profiling (native and custom)? - - Whatever LLVM supports, we support. I haven't seen any problems with standard tools. - - rr works with Servo. - - What about Windows? - - A lot of the Windows stuff is based on LLVM, whose PDB support is relatively immature. We're keeping up to date with LLVM to benefit from new improvements. - - Currently can't inspect local variables; sometimes stack traces are weird. - - vlad: Maybe use the new stuff? - - That uses the Clang front-end but uses MSVC++ for codegen; we can't use that for Rust. - - There are caveats. We emit DWARF and masquerade as C++. The fidelity is low. There are person-years of work to get full mature support. - - But it's good enough in practice for e.g. debugging Servo. - - We should validate all this; not just debugging but also profiling, crash reporting. - - If there are specific areas that need fixing, ask management for resources to do it. - - Rust team's priority right now is developer ergonomics and tooling. - - Diagnosing bugs reported "in the wild" - - Crash reporting (how do stack traces look; are symbols generated/stored properly) - - We got a reasonable stack trace from a crash report in a test run. - - How do we test the full workflow of crash reporter -> crash-stats.mozilla.com? - - bz: Introduce a crash behind a pref in nightly :-) - - How do we make sure bugs are fixed in both engines? - - upstream repo for create; changes land there first? - - Once we have real users, we have less flexibility to change things without regard for impact on them. - - This impacts code like rust-url or Servo layout code that's shared between Gecko and Servo (or other Rust projects) - - We have this problem with a lot of third-party code used in mozilla-central. - - We vendor everything. It makes our life easier in some ways, in others harder. - - Servo uses cargo rather than vendoring things. - - Maybe we can have a "cargo vendor" tool or similar that integrates with cargo to sync in-tree copies with upstream repos. - - Or a more general (not Rust-specific) tool for syncing/diffing m-c directories against their upstream sources. - - Servo has a mach command to do two-way sync between our vendored web-platform-tests and upstream. - - bholley: we need better tooling in general for getting fixes in deeply nested dependencies deployed to downstream code. - - Instead of one repo per crate, we could have one repo with many crates, if that helps. - - acrichto: Idiomatic Rust practice is the opposite of "vendor everything." It's much easier if you can find ways to work the Rust ecosystem way. - - ted: I understand that but I don't think we're going to go all the way there. There may be somewhere halfway. - - gps: People really want bi-directional multi-vcs multi-repository "syncing" - - Rust, Servo, media, ICU, WebRTC, web-platform-tests, /devtools, /browser, .... - - Quilt/MQ like local patches - - gps: devs can pull from crates.io. builders absolutely cannot hit crates.io. - - ms2ger: We want this for devs' local builds too so they don't need to be connected after downloading/cloning m-c. - - m-c tree integration - - Vendoring and updating crates.io/git dependencies - - Are there other constraints you can pass along? - - Build one huge crate that builds all the Gecko/Rust code into one world - - Not sure on the compilation model; need to clean it up a test rust-url example set of patches - - implicitly trust code review from Servo/Rust code / module policy concerns - - Common to 3rd party code. Rubber-stamp review? e.g., libjpeg... rely on push to try? - - Expect responsive upstream for servo code - - bisection - - simonsapin: Should we import commit history? - - gps: There are bots on FFOS that do automated commits - - It periodically pulls from upstream, and lands the new revision if it passes tests. So every few hours it's updated to the latest upstream - - ted: chromium does this, too via trybots - - gps: Can't just pull the full commit histroy because it might not be commit-level bisectable. Can roll upstream pushes into commits. - - Both Servo and mozilla-central are "push-level" bisectable but not commit-level. We try, but no automation enforcing that intermediate commits build. - - bz: Once we have Servo style code in Gecko we need to have process like "Intent to Ship" - + * Individual project status + * MooV parser + * is in-tree, linux 64, inboud on Mac. Next are Windows and Android + * Help on Windows, Android and B2G would be good! Bottleneck is CI. + * 230k executions in telemetry + * rust-url + * necko tests pass + * IDNA support currently being worked on (PR open) + * percent decoding of the hostname is quite different - until recently we didn't do it at all for hostnames, and now we only decode valid hostname characters (bug 412457) + * needed to make rust-url accept other schemes, such as resource:// in order to be able to run the browser, since some schemes also go through the parser (PR pending) + * Have not yet done a full 'try' run, which should flush remaining issues + * Did the build using cargo; still waiting for the rest of the build system integration + * What about BMP decoder? (bholley) + * nnethercote just rewrote in C++, but at least with a big test suite + * Isn't the MP4 parser bad? + * Have both of them in-tree, run both for now. So still fixing stagefright security problems. + * Platform support status - where are we at now? What blockers remain? + * Rust 1.5 stable just hit! + * Gives us multi-file crates, + * Dependency generation for incremental builds. + * jemalloc (fine to redirect) + * Can now redirect the Rust code to use the standard symbols in Gecko. nfroyd! + * work remaining for try builds + * rpath/origin support to avoid LD_LIBRARY_PATH in try + * Official rust binaries don't set LD_LIBRARY_PATH; we build a custom snapshot for the try servers + * status for each tier 1 platform + * Rust supports all tier 1 platforms; just need our build system integrations in space there (including OSX10.6, Linux64; Windows, Android, B2G need help. Just have to add more libstds) + * For some platforms like Android that just involves adding new targets to the build system and putting the additional libraries in place. Any volunteers? nfroyd! + * How much work would it take to make just the style system build on stable Rust (not require unstable nightly features)? + * pcwalton: Ms2ger has done work to move us off of non-necessary unstable features. Do you know how much unstable stuff layout is using? + * Ms2ger: Most things in the layout crate could be removed. + * Manishearth: I did an audit (https://gist.github.com/Manishearth/967bc78db5c751c38d03#layout, many of these have stabilized since) a few months ago. Most of the unstable features are left over from pre-1.0 code and we just haven't bothered changing them. Ignoring plugins, it's mostly easy. Plugins could be replaced with codegen. + * pcwalton: Do we even use any plugins in layout? + * (discussion to continue offline) + * Code / optimize / debug workflow + * How do we handle PGO build interactions (if in LLVM, can be added) + * We do PGO on Win (using MSVC) and Linux (GCC). + * I don't know what it means to do this. We use LLVM codegen across the board. As long as you treat us like other native libraries it should work out. + * The Rust code may not get the full benefit of PGO but it shouldn't cause problems. We currently compile some third-party code like ICU without PGO. + * LLVM has PGO support but we haven't done the legwork to use it. + * PGO is useful for devirtualization which isn't as important in Rust. + * It can also change optimizations based on hot vs. cold paths. + * bz: Once we have a style system in Rust, having PGO for the style system *would* be nice. + * How well does Rust work with the Gecko gdb/lldb/profiling (native and custom)? + * Whatever LLVM supports, we support. I haven't seen any problems with standard tools. + * rr works with Servo. + * What about Windows? + * A lot of the Windows stuff is based on LLVM, whose PDB support is relatively immature. We're keeping up to date with LLVM to benefit from new improvements. + * Currently can't inspect local variables; sometimes stack traces are weird. + * vlad: Maybe use the new stuff? + * That uses the Clang front-end but uses MSVC++ for codegen; we can't use that for Rust. + * There are caveats. We emit DWARF and masquerade as C++. The fidelity is low. There are person-years of work to get full mature support. + * But it's good enough in practice for e.g. debugging Servo. + * We should validate all this; not just debugging but also profiling, crash reporting. + * If there are specific areas that need fixing, ask management for resources to do it. + * Rust team's priority right now is developer ergonomics and tooling. + * Diagnosing bugs reported "in the wild" + * Crash reporting (how do stack traces look; are symbols generated/stored properly) + * We got a reasonable stack trace from a crash report in a test run. + * How do we test the full workflow of crash reporter -> crash-stats.mozilla.com? + * bz: Introduce a crash behind a pref in nightly :-) + * How do we make sure bugs are fixed in both engines? + * upstream repo for create; changes land there first? + * Once we have real users, we have less flexibility to change things without regard for impact on them. + * This impacts code like rust-url or Servo layout code that's shared between Gecko and Servo (or other Rust projects) + * We have this problem with a lot of third-party code used in mozilla-central. + * We vendor everything. It makes our life easier in some ways, in others harder. + * Servo uses cargo rather than vendoring things. + * Maybe we can have a "cargo vendor" tool or similar that integrates with cargo to sync in-tree copies with upstream repos. + * Or a more general (not Rust-specific) tool for syncing/diffing m-c directories against their upstream sources. + * Servo has a mach command to do two-way sync between our vendored web-platform-tests and upstream. + * bholley: we need better tooling in general for getting fixes in deeply nested dependencies deployed to downstream code. + * Instead of one repo per crate, we could have one repo with many crates, if that helps. + * acrichto: Idiomatic Rust practice is the opposite of "vendor everything." It's much easier if you can find ways to work the Rust ecosystem way. + * ted: I understand that but I don't think we're going to go all the way there. There may be somewhere halfway. + * gps: People really want bi-directional multi-vcs multi-repository "syncing" + * Rust, Servo, media, ICU, WebRTC, web-platform-tests, /devtools, /browser, .... + * Quilt/MQ like local patches + * gps: devs can pull from crates.io. builders absolutely cannot hit crates.io. + * ms2ger: We want this for devs' local builds too so they don't need to be connected after downloading/cloning m-c. + * m-c tree integration + * Vendoring and updating crates.io/git dependencies + * Are there other constraints you can pass along? + * Build one huge crate that builds all the Gecko/Rust code into one world + * Not sure on the compilation model; need to clean it up a test rust-url example set of patches + * implicitly trust code review from Servo/Rust code / module policy concerns + * Common to 3rd party code. Rubber-stamp review? e.g., libjpeg... rely on push to try? + * Expect responsive upstream for servo code + * bisection + * simonsapin: Should we import commit history? + * gps: There are bots on FFOS that do automated commits + * It periodically pulls from upstream, and lands the new revision if it passes tests. So every few hours it's updated to the latest upstream + * ted: chromium does this, too via trybots + * gps: Can't just pull the full commit histroy because it might not be commit-level bisectable. Can roll upstream pushes into commits. + * Both Servo and mozilla-central are "push-level" bisectable but not commit-level. We try, but no automation enforcing that intermediate commits build. + * bz: Once we have Servo style code in Gecko we need to have process like "Intent to Ship" Issues for the rust team - - Rust has integer overflow checking in debug builds, disabled in optimized builds. Is there any static proof of no overflow? - - No. - - In media we'd like overflow checking enabled all the time. - - Integer math in media is so fast compared to the overall video decoding process, we care much more about correctness than performance. - - You can enable this in optimized builds with a compiler flag. - - Would like to do it at the crate level - - In Rust, even if you get an overflow, it's usually less serious than C++ because it can't cause buffer overrun/underrun or similar safety issues. - - Array bounds checking is always enabled (in debug *and* release builds). - - Some feedback that rustc's error messages are not good enough. - - steveklabnik: Just recently the @RustLang twitter got two simultaneous messages saying "best error mesages ever!!!" and "terrible error messages! :(" - - We'd love specific feedback on any bad error messages. File issues. CC Niko and Manish - - Compile times. - - There are lots of tweaks you can do; we're always working on improving the compiler. - - Biggest improvement will be incremental compilation. Lots of active work going on but it'll take a long time (several months) to be landed. - - Our measurements show a steady improvement in compile speed over time, but there may be bad cases still. - - If you have crates with especially bad compile time, send them to us to profile. - - You can improve things by splitting into crates. - - But then you don't get inlining across them (except with functions explicitly marked #[inline]) - - (Also, your functions are currently round-robined / otherwise arbitrarily divided, so what does get inlined (if it's not #[inline]) can be unpredictable.) - - Servo team feels this pain a lot since we have some very very big crates that take minutes to compile, every time they change. - - cargo and external build tools - - incremental builds - - expect results maybe by summer 2016 - - New rust toolchain every 6 weeks? - - 150MB. 400MB uncompressed. - - Not good for places w/o reliable internet. - - There's some discussion about recycling (C++) build artifacts for Gecko, so e.g. frontend developers can build JS changes without needing a C++ compiler. - - Could leverage this for Rust too. - - Of course you'd still need to build X megabytes of pre-built binaries. - - At the beginning, we have little Rust code and frequent changes to the Rust toolchain, so faster to download the build artifacts. - - Later, maybe we'll have more Rust code and more stable toolchain, so trade-off may change. - - But we don't want to force upstream projects (e.g. Servo) to not track the latest Rust - - At this point, benefit of upgrading is usually to get perf improvements. Feature-wise we could be happy with stable Rust. - - Better to download Rust as source? - - No, can add hours to initial build time. - - We can just fedex a DVD with all the code every 6 weeks to contributors on slow networks. - - How long can we keep Rust as an optional dependency? - - Right now it's opt-in; eventually could be opt-out; at some point we'll have code that is Rust-only with no C++ fallback. - - We don't want to ship two URL parsers forever (especially if one is untested and they are diverging) - - This is a policy decision for the Build Config module owner - - Could download partial updates for rustc - - diff between releases currently likely to be quite large, possibly no improvement - + * Rust has integer overflow checking in debug builds, disabled in optimized builds. Is there any static proof of no overflow? + * No. + * In media we'd like overflow checking enabled all the time. + * Integer math in media is so fast compared to the overall video decoding process, we care much more about correctness than performance. + * You can enable this in optimized builds with a compiler flag. + * Would like to do it at the crate level + * In Rust, even if you get an overflow, it's usually less serious than C++ because it can't cause buffer overrun/underrun or similar safety issues. + * Array bounds checking is always enabled (in debug *and* release builds). + * Some feedback that rustc's error messages are not good enough. + * steveklabnik: Just recently the @RustLang twitter got two simultaneous messages saying "best error mesages ever!!!" and "terrible error messages! :(" + * We'd love specific feedback on any bad error messages. File issues. CC Niko and Manish + * Compile times. + * There are lots of tweaks you can do; we're always working on improving the compiler. + * Biggest improvement will be incremental compilation. Lots of active work going on but it'll take a long time (several months) to be landed. + * Our measurements show a steady improvement in compile speed over time, but there may be bad cases still. + * If you have crates with especially bad compile time, send them to us to profile. + * You can improve things by splitting into crates. + * But then you don't get inlining across them (except with functions explicitly marked #[inline]) + * (Also, your functions are currently round-robined / otherwise arbitrarily divided, so what does get inlined (if it's not #[inline]) can be unpredictable.) + * Servo team feels this pain a lot since we have some very very big crates that take minutes to compile, every time they change. + * cargo and external build tools + * incremental builds + * expect results maybe by summer 2016 + * New rust toolchain every 6 weeks? + * 150MB. 400MB uncompressed. + * Not good for places w/o reliable internet. + * There's some discussion about recycling (C++) build artifacts for Gecko, so e.g. frontend developers can build JS changes without needing a C++ compiler. + * Could leverage this for Rust too. + * Of course you'd still need to build X megabytes of pre-built binaries. + * At the beginning, we have little Rust code and frequent changes to the Rust toolchain, so faster to download the build artifacts. + * Later, maybe we'll have more Rust code and more stable toolchain, so trade-off may change. + * But we don't want to force upstream projects (e.g. Servo) to not track the latest Rust + * At this point, benefit of upgrading is usually to get perf improvements. Feature-wise we could be happy with stable Rust. + * Better to download Rust as source? + * No, can add hours to initial build time. + * We can just fedex a DVD with all the code every 6 weeks to contributors on slow networks. + * How long can we keep Rust as an optional dependency? + * Right now it's opt-in; eventually could be opt-out; at some point we'll have code that is Rust-only with no C++ fallback. + * We don't want to ship two URL parsers forever (especially if one is untested and they are diverging) + * This is a policy decision for the Build Config module owner + * Could download partial updates for rustc + * diff between releases currently likely to be quite large, possibly no improvement