Tuesday, February 4, 2014

Android web performance testing takes a big leap forward

We've been working to bring better support for measuring web performance on mobile for a while.  Michael Klepikov started out by building out a new cross-platform test agent for WebPagetest that runs on Node.js, can run WebDriver/Selenium scripts and can talk to the Dev Tools interface for Chrome.  Todd Wright  extended that support to talk to mobile Chrome on android and even Safari on iOS using a Dev Tools proxy that he created.  Browser support has been really good for a while and we could get great request data and full timelines but video has always been the blocker for being able to launch.  When Android 4.4 launched with the ability to record 60FPS video on-device with very low overhead it solved the last issue that was holding us back from launching.

WebPageTest now supports Chrome stable and Beta on Android 4.4

For private instances the code is all in github and once it has had a couple of weeks of public use and shaking through any issues I'll cut an official release.  If you want to try it out before then you'll need both the web and agent code to support the new video capture capabilities (agent setup instructions are here).

Live on the public instance are a collection of devices in the Dulles location for testing:



There are:

  • 5 Motorola G's
  • 2 Nexus 5's
  • 1 Nexus 7 in Portrait Mode
  • 1 Nexus 7 in Landscape Mode
To select the devices, just select the Dulles location from the location list and they will show up in the list of browsers.

All of the devices are also available through the API for automated testing with the location ID's available here.

For now all of the devices are using a fixed 3G connection profile but hopefully soon they will have support for arbitrary connection profiles as well.

The video capture on the mobile devices is significantly better than what we have on Desktop and I highly encourage you to try it out.  Most of the sites I have tried out take a surprisingly long time to display anything (one second is a good, aggressive target to shoot for).  Since the mobile devices support much faster capture than desktop, the filmstrip view in WebPageTest has a new 60FPS option for displaying every frame and being able to see EXACTLY when something was displayed.


The increased resolution really helps when aligning the video with what is happening in the waterfall.

We also get full dev tools timeline views of what is going on which is particularly important on mobile given the slower processing (timelines are captured automatically when video is enabled or optionally in the "Chrome" tab of the advanced settings otherwise).


If you're really adventurous you can also submit WebDriver/Selenium scripts for testing (though it hasn't had a lot of exercise so there may be issues).

Most of the test features that you are used to using on desktop still don't work but over the next few weeks we should be able to fill some of them in as well as add some more mobile-specific capabilities:
  • Packet Captures (tcpdump)
  • Arbitrary connection profiles
  • Testing with Chrome's Data Reduction Proxy enabled
  • Arbitrary Chrome command-line switches (will allow for DNS rewriting and cert ignoring)
  • Test sharding so individual tests can run in parallel across devices and complete faster
  • Storing of response bodies
  • Javascript disabling
  • SPOF testing
  • Basic WPT scripting support (logData, navigate and exec commands initially)
Take the devices for a spin and let us know if you see any issues.  If you don't see the devices online it's possible that the agent threw an exception that we didn't handle and I should be able to bring them back online pretty quickly (ping me if it looks like they've been offline for a while).

10 comments:

  1. Really useful, what an impressive direction to take webpagetest into!
    Small thing: 60 fps film strip view doesn't actually show the last frame (frames?) with rendered content.
    Look at http://www.webpagetest.org/video/compare.php?tests=140204_NH_ZZ8-r%3A1-c%3A0&thumbSize=200&ival=16.67&end=visual which seems to never render and compare with 0.1 s resolution.

    ReplyDelete
    Replies
    1. Thanks. Computer math can be funny sometimes. It should be fixed now.

      Delete
  2. hi Patrick,

    It will be very helpful for me if you can provide me the link to get the test agent for linux (even if it is not functionally complete or tested).

    Thanks in advance
    Ram

    ReplyDelete
    Replies
    1. Links are in the post in the first paragraph after the announcement (private instances). There's a link to the github repository as well as the agent setup instructions.

      Delete
    2. Thanks Patrick. The instructions include device config also .
      If I want to just have a linux agent ( my server is already on linux) I presume these instructions would be the same correct ?

      Delete
    3. Mobile or desktop? If you are doing desktop then you wouldn't do the device setup part.

      Delete
  3. Is there any documentation about getting the iOS agent working? It seems that all the code is there ready to go, but I just can't make it run. Any hint?

    ReplyDelete
    Replies
    1. No, not yet. iOS doesn't support video capture yet and the code is still a bit shaky so we wanted to stabilize it a bit first. You're welcome to play with it but it requires a jailbreak, a launcher app and the ios debug proxy so there are quite a few moving parts.

      Delete
  4. Can we test Native app performance ?

    ReplyDelete

All comments are moderated and may take a while to appear.