ci: Run bencher jobs on self-hosted runners (#39272)

some bencher jobs, specifically linux release profile jobs, measure
runtime perf by running speedometer and dromaeo. doing this on
GitHub-hosted runners is suboptimal, because GitHub-hosted runners are
not under our control and their performance can vary wildly depending on
what hosts we get and how busy they are.

this patch depends on #39270 and #39271, and the new
[ci3](https://ci3.servo.org/) and [ci4](https://ci4.servo.org/) servers
deployed in servo/ci-runners#49. these servers provide a more controlled
environment for benchmarking by using known hardware that runs one job
at a time and no other work (servo/project#160), with some of the
[techniques](https://github.com/servo/servo/wiki/Servo-Benchmarking-Report-(October-2024)#methodology)
[we’ve](https://github.com/servo/servo/wiki/Servo-Benchmarking-Report-(November-2024)#methodology)
[developed](https://github.com/servo/servo/wiki/Servo-Benchmarking-Report-%28December-2024%29#methodology)
for accurate measurement:
- [we disable CPU frequency boost and
hyperthreading](364719f210...5c02999bbc (diff-e6f17b25776ca26c2880cc3a4e3b99a0642ea968a8d6214763cb6467cc1251cfR239-R256))
- [we pin guest CPUs to specific host
CPUs](364719f210...5c02999bbc (diff-cdaac247bfd7d30f8c835083adab39c7ead8791802498285ea2ce9e023cc5f06R15-R26))
- [we isolate a subset of CPUs from all processes and scheduling
interrupts](364719f210...5c02999bbc (diff-e6f17b25776ca26c2880cc3a4e3b99a0642ea968a8d6214763cb6467cc1251cfR97-R102))
- the bencher workflow does not take advantage of this yet, but it will
in a later patch

to use ci3 and ci4 for bencher jobs, we add them to the list of
self-hosted runner servers, then make the bencher workflow try to find a
servo-ubuntu2204-bench runner if speedometer and/or dromaeo have been
requested. to avoid mixing data, we set the bencher “testbed” based on
where the runner came from:
- for GitHub-hosted runners, we continue to use “ubuntu-22.04”
- for runners on ci3 or ci4, we use
“self-hosted-image:[servo-ubuntu2204-bench](e911a23eff/profiles/servo-ubuntu2204-bench)”

Testing:
- before, always GitHub-hosted: [job
run](https://github.com/servo/servo/actions/runs/18276911520/job/52032330450)
→
[report](https://bencher.dev/perf/servo/reports/86aa60cc-9d42-418f-a639-07b8604b30fb)
- after, self-hosted: [job
run](https://github.com/servo/servo/actions/runs/18404778338/job/52442477058)
→
[report](https://bencher.dev/perf/servo/reports/6feed0ac-655a-4e17-9351-41cba8d283b2)
- after, GitHub-hosted: [job
run](https://github.com/servo/servo/actions/runs/18404806546/job/52442697457)
→
[report](https://bencher.dev/perf/servo/reports/235a4ee0-340d-458b-9be4-953568b0923d)
- there are also counterparts for other platforms in the workflow runs
above

Fixes: #39269

---------

Signed-off-by: Delan Azabani <dazabani@igalia.com>

.github/actions/runner-select/action.yml

@@ -26,6 +26,8 @@ outputs:
     value: ${{ steps.select.outputs.unique_id }}
   selected-runner-label:
     value: ${{ steps.select.outputs.selected_runner_label }}
+  runner-type-label:
+    value: ${{ steps.select.outputs.runner_type_label }}
   is-self-hosted:
     value: ${{ steps.select.outputs.is_self_hosted }}
 
@@ -48,6 +50,7 @@ runs:
         fall_back_to_github_hosted() {
           echo 'Falling back to GitHub-hosted runner'
           echo "selected_runner_label=$github_hosted_runner_label" | tee -a $GITHUB_OUTPUT
+          echo "runner_type_label=$github_hosted_runner_label" | tee -a $GITHUB_OUTPUT
           echo 'is_self_hosted=false' | tee -a $GITHUB_OUTPUT
           exit 0
         }
@@ -76,6 +79,8 @@ runs:
           https://ci0.servo.org \
           https://ci1.servo.org \
           https://ci2.servo.org \
+          https://ci3.servo.org \
+          https://ci4.servo.org \
         | shuf); do
           # Use the monitor API to reserve a runner. If we get an object with
           # runner details, we succeeded. If we get null, we failed.
@@ -88,6 +93,7 @@ runs:
               && jq -e . $result > /dev/null; then
             echo
             echo "selected_runner_label=reserved-for:$unique_id" | tee -a $GITHUB_OUTPUT
+            echo "runner_type_label=self-hosted-image:$self_hosted_image_name" | tee -a $GITHUB_OUTPUT
             echo 'is_self_hosted=true' | tee -a $GITHUB_OUTPUT
             exit 0
           fi

.github/workflows/bencher.yml

@@ -34,6 +34,10 @@ on:
         required: false
         default: false
         type: boolean
+      force-github-hosted-runner:
+        required: false
+        type: boolean
+        default: false
 
 env:
   RUST_BACKTRACE: 1
@@ -42,22 +46,76 @@ env:
   BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT || 'servo' }}
 
 jobs:
-  bencher:
-    name: Bencher (${{ inputs.target }})
-    # This needs to be kept in sync with the `--testbed` argument sent to bencher.
+  # Runs the underlying job (“workload”) on a self-hosted runner if available,
+  # with the help of a `runner-select` job and a `runner-timeout` job.
+  runner-select:
+    runs-on: ubuntu-22.04
+    outputs:
+      unique-id: ${{ steps.select.outputs.unique-id }}
+      selected-runner-label: ${{ steps.select.outputs.selected-runner-label }}
+      runner-type-label: ${{ steps.select.outputs.runner-type-label }}
+      is-self-hosted: ${{ steps.select.outputs.is-self-hosted }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          sparse-checkout: '.github'
+      - name: Runner select
+        id: select
+        uses: ./.github/actions/runner-select
+        with:
+          monitor-api-token: ${{ secrets.SERVO_CI_MONITOR_API_TOKEN }}
+          github-hosted-runner-label: ubuntu-22.04
+          self-hosted-image-name: servo-ubuntu2204-bench
+          # You can disable self-hosted runners globally by creating a repository variable named
+          # NO_SELF_HOSTED_RUNNERS with any non-empty value.
+          # <https://github.com/servo/servo/settings/variables/actions>
+          NO_SELF_HOSTED_RUNNERS: ${{ vars.NO_SELF_HOSTED_RUNNERS }}
+          # Any other boolean conditions that disable self-hosted runners go here.
+          # No need to use self-hosted runners if we’re only measuring binary size.
+          force-github-hosted-runner: ${{ !(inputs.speedometer || inputs.dromaeo) || inputs.force-github-hosted-runner }}
+  runner-timeout:
+    needs:
+      - runner-select
+    if: ${{ fromJSON(needs.runner-select.outputs.is-self-hosted) }}
     runs-on: ubuntu-22.04
     steps:
-      - uses: actions/checkout@v4
-        if: github.event_name != 'pull_request_target'
+      - name: Checkout
+        uses: actions/checkout@v4
         with:
-          fetch-depth: 0
+          sparse-checkout: '.github'
+      - name: Runner timeout
+        uses: ./.github/actions/runner-timeout
+        with:
+          github_token: '${{ secrets.GITHUB_TOKEN }}'
+          unique-id: '${{ needs.runner-select.outputs.unique-id }}'
+
+  bencher:
+    needs:
+      - runner-select
+    name: Bencher (${{ inputs.target }}) [${{ needs.runner-select.outputs.unique-id }}]
+    runs-on: ${{ needs.runner-select.outputs.selected-runner-label }}
+    steps:
+      - uses: actions/checkout@v4
+        if: ${{ runner.environment != 'self-hosted' && github.event_name != 'pull_request_target' }}
       # This is necessary to checkout the pull request if this run was triggered via a
       # `pull_request_target` event.
       - uses: actions/checkout@v4
-        if: github.event_name == 'pull_request_target'
+        if: ${{ runner.environment != 'self-hosted' && github.event_name == 'pull_request_target' }}
         with:
-          ref: refs/pull/${{ github.event.number }}/head
-          fetch-depth: 0
+          ref: ${{ github.event.pull_request.head.sha }}
+      # Faster checkout for self-hosted runner that uses prebaked repo.
+      - if: ${{ runner.environment == 'self-hosted' && github.event_name != 'pull_request_target' }}
+        run: git fetch --depth=1 origin $GITHUB_SHA
+      - if: ${{ runner.environment == 'self-hosted' && github.event_name == 'pull_request_target' }}
+        run: git fetch --depth=1 origin ${{ github.event.pull_request.head.sha }}
+      - if: ${{ runner.environment == 'self-hosted' }}
+        # Same as `git switch --detach FETCH_HEAD`, but fixes up dirty working
+        # trees, in case the runner image was baked with a dirty working tree.
+        run: |
+          git switch --detach
+          git reset --hard FETCH_HEAD
+
       - uses: actions/download-artifact@v4
         with:
           name: ${{ inputs.profile }}-binary-${{ inputs.target }}
@@ -70,9 +128,10 @@ jobs:
         if: ${{ inputs.compressed-file-path != '' && !contains(inputs.compressed-file-path, '.tar.gz') }}
         run: unzip ${{ inputs.compressed-file-path }}
       - name: Setup Python
+        if: ${{ runner.environment != 'self-hosted' }}
         uses: ./.github/actions/setup-python
       - name: Bootstrap dependencies
-        if: ${{ inputs.speedometer == true || inputs.dromaeo == true }}
+        if: ${{ runner.environment != 'self-hosted' && (inputs.speedometer || inputs.dromaeo) }}
         run: |
           sudo apt update
           sudo apt install -qy --no-install-recommends mesa-vulkan-drivers
@@ -127,7 +186,7 @@ jobs:
         if: ${{ github.event_name == 'push' && github.ref_name == 'try' }}
         run: |
             git remote add upstream https://github.com/servo/servo
-            git fetch upstream main
+            git fetch --unshallow upstream main
             echo "RUN_BENCHER_OPTIONS=--branch try \
             --github-actions ${{ secrets.GITHUB_TOKEN }} \
             --hash $(git rev-parse HEAD~1) \
@@ -140,6 +199,7 @@ jobs:
         continue-on-error: true
         run: |
           ./etc/ci/bencher.py merge ${{ env.SERVO_FILE_SIZE_RESULT }} ${{ env.SERVO_STRIPPED_FILE_SIZE_RESULT }} ${{ env.SERVO_SPEEDOMETER_RESULT }} ${{ env.SERVO_DROMAEO_RESULT }} --bmf-output b.json
+          testbed='${{ needs.runner-select.outputs.runner-type-label }}'
           bencher run --adapter json --file b.json \
-            --project ${{ env.BENCHER_PROJECT }} --token ${{ secrets.BENCHER_API_TOKEN }} --testbed ubuntu-22.04 \
+            --project ${{ env.BENCHER_PROJECT }} --token ${{ secrets.BENCHER_API_TOKEN }} --testbed "$testbed" \
             $RUN_BENCHER_OPTIONS

.github/workflows/linux.yml

@@ -103,6 +103,7 @@ jobs:
     outputs:
       unique-id: ${{ steps.select.outputs.unique-id }}
       selected-runner-label: ${{ steps.select.outputs.selected-runner-label }}
+      runner-type-label: ${{ steps.select.outputs.runner-type-label }}
       is-self-hosted: ${{ steps.select.outputs.is-self-hosted }}
     steps:
       - name: Checkout

.github/workflows/mac.yml

@@ -89,6 +89,7 @@ jobs:
     outputs:
       unique-id: ${{ steps.select.outputs.unique-id }}
       selected-runner-label: ${{ steps.select.outputs.selected-runner-label }}
+      runner-type-label: ${{ steps.select.outputs.runner-type-label }}
       is-self-hosted: ${{ steps.select.outputs.is-self-hosted }}
     steps:
       - name: Checkout

.github/workflows/windows.yml

@@ -80,6 +80,7 @@ jobs:
     outputs:
       unique-id: ${{ steps.select.outputs.unique-id }}
       selected-runner-label: ${{ steps.select.outputs.selected-runner-label }}
+      runner-type-label: ${{ steps.select.outputs.runner-type-label }}
       is-self-hosted: ${{ steps.select.outputs.is-self-hosted }}
     steps:
       - name: Checkout

python/servo/testing_commands.py

@@ -499,15 +499,19 @@ class MachCommands(CommandBase):
     @Command("test-dromaeo", description="Run the Dromaeo test suite", category="testing")
     @CommandArgument("tests", default=["recommended"], nargs="...", help="Specific tests to run")
     @CommandArgument("--bmf-output", default=None, help="Specify BMF JSON output file")
-    @CommandBase.common_command_arguments(binary_selection=True)
-    def test_dromaeo(self, tests: list[str], servo_binary: str, bmf_output: str | None = None) -> None:
-        return self.dromaeo_test_runner(tests, servo_binary, bmf_output)
+    @CommandBase.common_command_arguments(build_type=True, binary_selection=True)
+    def test_dromaeo(
+        self, tests: list[str], build_type: BuildType, servo_binary: str, bmf_output: str | None = None, **kwargs: Any
+    ) -> None:
+        return self.dromaeo_test_runner(tests, servo_binary, bmf_output, build_type.profile)
 
     @Command("test-speedometer", description="Run servo's speedometer", category="testing")
     @CommandArgument("--bmf-output", default=None, help="Specify BMF JSON output file")
-    @CommandBase.common_command_arguments(binary_selection=True)
-    def test_speedometer(self, servo_binary: str, bmf_output: str | None = None) -> None:
-        return self.speedometer_runner(servo_binary, bmf_output)
+    @CommandBase.common_command_arguments(build_type=True, binary_selection=True)
+    def test_speedometer(
+        self, build_type: BuildType, servo_binary: str, bmf_output: str | None = None, **kwargs: Any
+    ) -> None:
+        return self.speedometer_runner(servo_binary, bmf_output, build_type.profile)
 
     @Command("test-speedometer-ohos", description="Run servo's speedometer on a ohos device", category="testing")
     @CommandArgument("--bmf-output", default=None, help="Specifcy BMF JSON output file")
@@ -625,7 +629,7 @@ class MachCommands(CommandBase):
 
         return call([run_file, cmd, bin_path, base_dir])
 
-    def dromaeo_test_runner(self, tests: list[str], binary: str, bmf_output: str | None) -> None:
+    def dromaeo_test_runner(self, tests: list[str], binary: str, bmf_output: str | None, profile: str) -> None:
         base_dir = path.abspath(path.join("tests", "dromaeo"))
         dromaeo_dir = path.join(base_dir, "dromaeo")
         run_file = path.join(base_dir, "run_dromaeo.py")
@@ -649,7 +653,12 @@ class MachCommands(CommandBase):
         # Check that a release servo build exists
         bin_path = path.abspath(binary)
 
-        return check_call([run_file, "|".join(tests), bin_path, base_dir, bmf_output])
+        args = [run_file, "|".join(tests), bin_path, base_dir]
+        if bmf_output is not None:
+            args.append(bmf_output)
+            args.append(profile)
+
+        return check_call(args)
 
     def speedometer_to_bmf(self, speedometer: dict[str, Any], bmf_output: str, profile: str | None = None) -> None:
         output = dict()
@@ -683,7 +692,7 @@ class MachCommands(CommandBase):
         with open(bmf_output, "w", encoding="utf-8") as f:
             json.dump(output, f, indent=4)
 
-    def speedometer_runner(self, binary: str, bmf_output: str | None) -> None:
+    def speedometer_runner(self, binary: str, bmf_output: str | None, profile: str) -> None:
         output = subprocess.check_output(
             [
                 binary,
@@ -707,7 +716,7 @@ class MachCommands(CommandBase):
         print(f"Score: {speedometer['Score']['mean']} ± {speedometer['Score']['delta']}")
 
         if bmf_output:
-            self.speedometer_to_bmf(speedometer, bmf_output)
+            self.speedometer_to_bmf(speedometer, bmf_output, profile)
 
     def speedometer_runner_ohos(self, bmf_output: str | None, profile: str | None) -> None:
         hdc_path = shutil.which("hdc")

tests/dromaeo/run_dromaeo.py

@@ -26,7 +26,7 @@ def run_servo(servo_exe, tests):
 
 # Print usage if command line args are incorrect
 def print_usage():
-    print("USAGE: {0} tests servo_binary dromaeo_base_dir [BMF JSON output]".format(sys.argv[0]))
+    print("USAGE: {0} tests servo_binary dromaeo_base_dir [<BMF JSON output> <Cargo profile>]".format(sys.argv[0]))
 
 
 post_data = None
@@ -48,13 +48,15 @@ class RequestHandler(SimpleHTTPRequestHandler):
 
 
 if __name__ == '__main__':
-    if len(sys.argv) == 4 or len(sys.argv) == 5:
+    if len(sys.argv) == 4 or len(sys.argv) == 6:
         tests = sys.argv[1]
         servo_exe = sys.argv[2]
         base_dir = sys.argv[3]
         bmf_output = ""
-        if len(sys.argv) == 5:
+        cargo_profile_prefix = ""
+        if len(sys.argv) == 6:
             bmf_output = sys.argv[4]
+            cargo_profile_prefix = f"{sys.argv[5]}/"
         os.chdir(base_dir)
 
         # Ensure servo binary can be found
@@ -82,7 +84,7 @@ if __name__ == '__main__':
         if bmf_output:
             output = dict()
             for (k, v) in data.items():
-                output[f"Dromaeo/{k}"] = {'throughput': {'value': float(v)}}
+                output[f"{cargo_profile_prefix}Dromaeo/{k}"] = {'throughput': {'value': float(v)}}
             with open(bmf_output, 'w', encoding='utf-8') as f:
                 json.dump(output, f, indent=4)
         proc.kill()