major mods for cmdline params, one script for desktop/android, etc. …

README.md

@@ -1,36 +1,77 @@
-# browsertime_on_android_scripts
-Scripts to run tests with browsertime on android
+# browsertime_scripts
 
-(e.g. geckoview_example, fennec, fenix)
+Scripts to run local performance tests with browsertime.
 
+Supports Firefox and Chrome desktop and android apps (e.g. geckoview_example, fenix).
 
-• Clone https://github.com/mozilla/browsertime
+## Usage ##
+
+• Clone the browsertime repo, https://github.com/sitespeedio/browsertime
 
 • Download the latest `geckodriver` release: https://github.com/mozilla/geckodriver/releases
 
-• Clone this repo, `chmod +x` the .sh scripts
+• Clone this repo, and  `chmod +x` the `.sh` scripts
 
-• Configure `run_android.py`:
+• Configure `run_test.py` to point to your browsertime and geckodriver paths (or specify them with `--browsertime` and `--geckodriver` )
 
-    host_ip (if using WebPageReplay)
-
-    android_serial
-
-    geckodriver_path (just downloaded)
-
-    browsertime_bin (should point within the newly cloned browsertime repo)
-
-    iterations
-
-    variants (experiment name, app, driving script, firefox preferences)
+        • browsertime_bin (point to bin/browsertime.js within the newly cloned browsertime repo)
+        • geckodriver_path (point to the geckodriver binary you just downloaded)
+
+• Configure the browser variants you'd like to compare. These can be seperate binaries or else preferences.
+
+  See `variants_android.py` or `variants_desktop.py` for examples.
 
-• Add the sites you'd like tested to `sites.txt`
+  For instance, to compare fenix without and with HTTP/3 enabled, you may do something like this:
 
-• Run with `python run_android.py`
+        variants = [('fenix', 'fenix.sh', 'org.mozilla.fenix', fenix_nightly_location, ''),
+                    ('fenix_h3', 'fenix.sh', 'org.mozilla.fenix', fenix_nightly_location, ' --firefox.preference network.http.http3.enabled:true' )]
+
+• Configure the sites you'd like tested in `sites.txt` or create a new sites text file and specify `--sites my_sites.txt`
 
------------------------
+• Run with `python run_test.py`
+
+• Additional options:
+
+          -h, --help            show this help message and exit
+          --debug               pass debugging flags to browsertime (-vvv)
+          -i ITERATIONS, -n ITERATIONS, --iterations ITERATIONS
+                                Number of iterations
+          -s SITES, --sites SITES
+                                File of sites
+          --serial SERIAL       Android serial #
+          --desktop             Desktop or mobile
+          --binarypath BINARYPATH
+                                path to firefox binary (desktop)
+          --webrender           Enable webrender (default: disabled)
+          --fullscreen          Run test in full-screen (Desktop)
+          --profile             Enable profiling
+          --visualmetrics       Calculate visualmetrics (SI/CSI/PSI)
+          --wpr_host_ip WPR_HOST_IP
+                                WebPageReplay host IP (optional)
+          --reload              test reload of the URL
+          --condition           load a site before the target URL
+          --prefs PREFS         Firefox preferences to use for all runs
+          --variants VARIANTS   python module with the variants defined, e.g.
+                                variants_android
+          --geckodriver GECKODRIVER
+                                path to geckodriver
+          --browsertime BROWSERTIME
+                                path to browsertime/bin/browsertime.js
+          --output_path OUTPUT_PATH
+                                Path for the browsertime json (defaults to
+                                'browsertime-results')
+          --restart_adb         Restart adb between variants (workaround to adb file
+                                descriptor leak)
+
+## Examples ##
+
+• Run on Desktop, with variants defined in a new file, `variants_fission.py`
+
+    python run_test.py --desktop --variants variants_fission
+• Run on android, load each site 10 times, and add the given additional prefs for each run
+
+    python run_test.py -n 10 --sites sites_1.txt --serial 89PX0DD5W --prefs "nglayout.debug.paint_flashing:true network.http.speculative-parallel-limit:7"
+
+• Run on Desktop, specifying the binary path
 
-## Cold Start Applink tests (w.i.p)
-• Uncomment [these lines](https://github.com/acreskeyMoz/browsertime_on_android_scripts
-/blob/4f15056b96a9db5be5c4f6807cfcd98fc87c4c30/run_android.py#L70-L73), and also be sure to disabled visual metric and video
-• Results are logged `(navigationStartTime - processStartTime)`, and also stored in the json as `browserScripts.pageinfo.processTimeShift`
+    python run_test.py --desktop --binarypath "/Applications/Firefox Nightly.app/Contents/MacOS/firefox"

clear_apps.sh

@@ -2,14 +2,12 @@
 
 apps=("org.mozilla.firefox" \
       "org.mozilla.geckoview_example" \
-      "org.mozilla.fenix" \
-      "org.mozilla.fenix.beta" \
-      "org.mozilla.fenix.performancetest" \
-      "org.mozilla.fennec_aurora")
+      "org.mozilla.fenix" )
 
+export adb_bin=/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb
 for app in ${apps[@]}; do
-  echo adb -s $ANDROID_SERIAL shell am force-stop $app
-  adb -s $ANDROID_SERIAL shell am force-stop $app
-  echo adb -s $ANDROID_SERIAL shell pm clear $app
-  adb -s $ANDROID_SERIAL shell pm clear $app
+  echo $adb_bin -s $ANDROID_SERIAL shell am force-stop $app
+  $adb_bin -s $ANDROID_SERIAL shell am force-stop $app
+  echo $adb_bin -s $ANDROID_SERIAL shell pm clear $app
+  $adb_bin -s $ANDROID_SERIAL shell pm clear $app
 done

desktop.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# N.B.: yargs doesn't parse `--firefox.android.intentArgument --ez`
+# properly, so always use `=--ez`!
+
+BASE='/home/jesup/src/mozilla/browsertime_on_android_scripts/'
+REPO='/home/jesup/src/mozilla/pageload/'
+
+if test $PERF; then
+    SCRIPT=${BASE}perf.sh
+else
+    SCRIPT=$FIREFOX_BINARY_PATH
+fi
+echo ./mach browsertime \
+    --firefox.binaryPath=\'"$SCRIPT"\' \
+  "$@"
+
+./mach browsertime \
+    --firefox.binaryPath=\'"$SCRIPT"\' \
+  "$@"

fenix.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+export adb_bin="~/.mozbuild/android-sdk-linux/platform-tools/adb"
+
 bash ./clear_apps.sh
 
 if [[ -n $ANDROID_SERIAL ]] ; then
@@ -24,7 +26,8 @@ $BROWSERTIME_BIN \
     --firefox.android.intentArgument=-d \
     --firefox.android.intentArgument="$LAUNCH_URL" \
     --browser firefox \
+    --firefox.binaryPath=/tmp/foo \
     $DEVICE_SERIAL_ARGS \
     "$@"
 
-adb -s $ANDROID_SERIAL shell am force-stop $PACKAGE
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fenix_no_clear.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+
+#: PACKAGE ${PACKAGE:=org.mozilla.firefox}
+: PACKAGE ${PACKAGE:=org.mozilla.fenix.performancetest}
+
+export adb_bin="/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb"
+
+if [[ -n $ANDROID_SERIAL ]] ; then
+    DEVICE_SERIAL_ARGS="--firefox.android.deviceSerial=$ANDROID_SERIAL --chrome.android.deviceSerial=$ANDROID_SERIAL"
+else
+    DEVICE_SERIAL_ARGS=
+fi
+
+export BROWSERTIME_BIN=tools/browsertime/node_modules/browsertime/bin/browsertime.js
+
+# N.B.: yargs doesn't parse `--firefox.android.intentArgument --ez`
+# properly, so always use `=--ez`!
+$BROWSERTIME_BIN \
+    --android \
+    --skipHar \
+    --firefox.geckodriverPath="/home/jesup/src/mozilla/browsertime_on_android_scripts/geckodriver" \
+    --firefox.android.package "$PACKAGE" \
+    --firefox.android.activity "org.mozilla.fenix.IntentReceiverActivity" \
+    --firefox.android.intentArgument=-a \
+    --firefox.android.intentArgument=android.intent.action.VIEW \
+    --firefox.android.intentArgument=-d \
+    --firefox.android.intentArgument="data:," \
+    --browser firefox \
+    --firefox.binaryPath=/tmp/foo \
+    --firefox.android.intentArgument=--ez \
+    --firefox.android.intentArgument=performancetest \
+    --firefox.android.intentArgument=true \
+    --firefox.geckoProfiler true --firefox.geckoProfilerParams.interval 5  --firefox.geckoProfilerParams.features "js,stackwalk,leaf" --firefox.geckoProfilerParams.threads "GeckoMain,socket,url,ava,cert,html" \
+    --pageCompleteWaitTime 10000 \
+    -n 3 \
+    --resultDir "browsertime-results/remote_settings_condprof/" \
+    --browsertime.url https://www.google.com/ \
+    --firefox.profileTemplate=/home/jesup/src/mozilla/browsertime_on_android_scripts/fenix.profile \
+    --firefox.preference services.settings.loglevel:debug \
+    https://www.google.com/ 
+
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fenix_performancetest.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+: PACKAGE ${PACKAGE:=org.mozilla.fenix.performancetest}
+
+bash /home/jesup/src/mozilla/browsertime_on_android_scripts/clear_apps.sh
+export adb_bin="/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb"
+
+if [[ -n $ANDROID_SERIAL ]] ; then
+    DEVICE_SERIAL_ARGS="--firefox.android.deviceSerial=$ANDROID_SERIAL --chrome.android.deviceSerial=$ANDROID_SERIAL"
+else
+    DEVICE_SERIAL_ARGS=
+fi
+
+# N.B.: yargs doesn't parse `--firefox.android.intentArgument --ez`
+# properly, so always use `=--ez`!
+$BROWSERTIME_BIN \
+    --android \
+    --skipHar \
+    --firefox.geckodriverPath="$GECKODRIVER_PATH" \
+    --firefox.android.package "$PACKAGE" \
+    --firefox.android.activity "org.mozilla.fenix.IntentReceiverActivity" \
+    --firefox.android.intentArgument=-a \
+    --firefox.android.intentArgument=android.intent.action.VIEW \
+    --firefox.android.intentArgument=-d \
+    --firefox.android.intentArgument="data:," \
+    --browser firefox \
+    --firefox.binaryPath=/tmp/foo \
+    $DEVICE_SERIAL_ARGS \
+    "$@"
+
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fenix_performancetest_condprof.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+: PACKAGE ${PACKAGE:=org.mozilla.fenix.performancetest}
+
+bash /home/jesup/src/mozilla/browsertime_on_android_scripts/clear_apps.sh
+export adb_bin="/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb"
+
+if [[ -n $ANDROID_SERIAL ]] ; then
+    DEVICE_SERIAL_ARGS="--firefox.android.deviceSerial=$ANDROID_SERIAL --chrome.android.deviceSerial=$ANDROID_SERIAL"
+else
+    DEVICE_SERIAL_ARGS=
+fi
+
+# N.B.: yargs doesn't parse `--firefox.android.intentArgument --ez`
+# properly, so always use `=--ez`!
+$BROWSERTIME_BIN \
+    --android \
+    --skipHar \
+    --firefox.geckodriverPath="$GECKODRIVER_PATH" \
+    --firefox.android.package "$PACKAGE" \
+    --firefox.android.activity "org.mozilla.fenix.IntentReceiverActivity" \
+    --firefox.android.intentArgument=-a \
+    --firefox.android.intentArgument=android.intent.action.VIEW \
+    --firefox.android.intentArgument=-d \
+    --firefox.android.intentArgument="data:," \
+    --browser firefox \
+    --firefox.binaryPath=/tmp/foo \
+    --firefox.profileTemplate=/home/jesup/src/mozilla/browsertime_on_android_scripts/fenix.profile \
+    $DEVICE_SERIAL_ARGS \
+    "$@"
+
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fenix_perf.sh → fenix_release.sh

@@ -1,6 +1,9 @@
 #!/bin/bash
 
-bash ./clear_apps.sh
+: PACKAGE ${PACKAGE:=org.mozilla.firefox}
+
+bash /home/jesup/src/mozilla/browsertime_on_android_scripts/clear_apps.sh
+export adb_bin="/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb"
 
 if [[ -n $ANDROID_SERIAL ]] ; then
     DEVICE_SERIAL_ARGS="--firefox.android.deviceSerial=$ANDROID_SERIAL --chrome.android.deviceSerial=$ANDROID_SERIAL"
@@ -15,16 +18,17 @@ $BROWSERTIME_BIN \
     --skipHar \
     --firefox.geckodriverPath="$GECKODRIVER_PATH" \
     --firefox.android.package "$PACKAGE" \
-    --firefox.android.activity "org.mozilla.fenix.browser.BrowserPerformanceTestActivity" \
+    --firefox.android.activity "org.mozilla.fenix.IntentReceiverActivity" \
     --firefox.android.intentArgument=-a \
     --firefox.android.intentArgument=android.intent.action.VIEW \
+    --firefox.android.intentArgument=-d \
+    --firefox.android.intentArgument="data:," \
+    --browser firefox \
+    --firefox.binaryPath=/tmp/foo \
     --firefox.android.intentArgument=--ez \
     --firefox.android.intentArgument=performancetest \
     --firefox.android.intentArgument=true \
-    --firefox.android.intentArgument=-d \
-    --firefox.android.intentArgument="$LAUNCH_URL" \
-    --browser firefox \
     $DEVICE_SERIAL_ARGS \
     "$@"
 
-adb -s $ANDROID_SERIAL shell am force-stop $PACKAGE
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fenix_release_condprof.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+: PACKAGE ${PACKAGE:=org.mozilla.firefox}
+
+bash /home/jesup/src/mozilla/browsertime_on_android_scripts/stop_apps.sh
+export adb_bin="/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb"
+
+if [[ -n $ANDROID_SERIAL ]] ; then
+    DEVICE_SERIAL_ARGS="--firefox.android.deviceSerial=$ANDROID_SERIAL --chrome.android.deviceSerial=$ANDROID_SERIAL"
+else
+    DEVICE_SERIAL_ARGS=
+fi
+
+# N.B.: yargs doesn't parse `--firefox.android.intentArgument --ez`
+# properly, so always use `=--ez`!
+$BROWSERTIME_BIN \
+    --android \
+    --skipHar \
+    --firefox.geckodriverPath="$GECKODRIVER_PATH" \
+    --firefox.android.package "$PACKAGE" \
+    --firefox.android.activity "org.mozilla.fenix.IntentReceiverActivity" \
+    --firefox.android.intentArgument=-a \
+    --firefox.android.intentArgument=android.intent.action.VIEW \
+    --firefox.android.intentArgument=-d \
+    --firefox.android.intentArgument="data:," \
+    --firefox.profileTemplate=/home/jesup/src/mozilla/browsertime_on_android_scripts/fenix.profile \
+    --browser firefox \
+    --firefox.binaryPath=/tmp/foo \
+    --firefox.android.intentArgument=--ez \
+    --firefox.android.intentArgument=performancetest \
+    --firefox.android.intentArgument=true \
+    $DEVICE_SERIAL_ARGS \
+    "$@"
+
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fennec.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
 
+export adb_bin="~/.mozbuild/android-sdk-linux/platform-tools/adb"
 bash ./clear_apps.sh
 
 if [[ -n $ANDROID_SERIAL ]] ; then
@@ -24,7 +25,8 @@ $BROWSERTIME_BIN \
     --firefox.android.intentArgument=skipstartpane                  \
     --firefox.android.intentArgument=true                           \
     --browser firefox                                               \
+    --firefox.binaryPath=/tmp/foo \
     $DEVICE_SERIAL_ARGS                                             \
   "$@"
 
-adb -s $ANDROID_SERIAL shell am force-stop $PACKAGE
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE

fennec_condprof.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+: PACKAGE ${PACKAGE:=org.mozilla.firefox}
+
+bash /home/jesup/src/mozilla/browsertime_on_android_scripts/stop_apps.sh
+export adb_bin="/home/jesup/.mozbuild/android-sdk-linux/platform-tools/adb"
+
+if [[ -n $ANDROID_SERIAL ]] ; then
+    DEVICE_SERIAL_ARGS="--firefox.android.deviceSerial=$ANDROID_SERIAL --chrome.android.deviceSerial=$ANDROID_SERIAL"
+else
+    DEVICE_SERIAL_ARGS=
+fi
+
+# N.B.: yargs doesn't parse `--firefox.android.intentArgument --ez`
+# properly, so always use `=--ez`!
+$BROWSERTIME_BIN \
+    --android \
+    --skipHar \
+    --firefox.geckodriverPath="$GECKODRIVER_PATH"                   \
+    --firefox.android.package "$PACKAGE"                            \
+    --firefox.android.activity 'org.mozilla.gecko.BrowserApp'       \
+    --firefox.android.intentArgument=-a                             \
+    --firefox.android.intentArgument=android.intent.action.VIEW     \
+    --firefox.android.intentArgument=-d                             \
+    --firefox.android.intentArgument="data:,"                       \
+    --firefox.android.intentArgument=--ez                           \
+    --firefox.android.intentArgument=skipstartpane                  \
+    --firefox.android.intentArgument=true                           \
+    --browser firefox                                               \
+    --firefox.binaryPath=/tmp/foo \
+    --firefox.profileTemplate=/home/jesup/src/mozilla/browsertime_on_android_scripts/fennec.profile \
+    $DEVICE_SERIAL_ARGS                                             \
+  "$@"
+
+$adb_bin -s $ANDROID_SERIAL shell am force-stop $PACKAGE