Measuring The Washington Post’s Progressive Web App

Written by Jason Grigsby on

Yesterday, I examined how The Washington Post’s Progressive Web App performs on iOS. Today, I want to take a look behind the scenes on the difficulties I had conducting that analysis.

Instrumenting The Washington Post

Often when organizations make big changes to their site, it is difficult to compare the old site to the new site unless you work for the organization and have access to their development servers.

Fortunately, the Washington Post hasn’t rolled out the Progressive Web App to all of their users yet—it is currently only being served to 10% of visitors. So most people are still seeing their old m-dot site.

You can opt in to the Progressive Web App by visiting on a mobile device. This meant that I could compare the behavior on iOS of the Progressive Web App versus the old mobile site.

It can be difficult to tell the difference between the old mobile site and the Progressive Web App because they are visually similar. One tell-tale sign that you’re looking at the Progressive Web App is the load time indicator present under the byline.

The web page on the left shows The Washington Post's old mobile site. The web page on the right is the new Progressive Web App. The text 'Loaded in 0.26 seconds' is the tell.

If you see the load time, you can be confident that you’re looking at the Progressive Web App. Even if you visit the opt-in URL, if you don’t see the indicator, it may mean that you’ve navigated too quickly and the PWA’s application logic hasn’t taken over navigation between pages yet.

Measuring a Progressive Web App is tricky

I started digging into the Washington Post’s Progressive Web App where I usually start—using WebPageTest. It is my go-to tool when it comes to measuring performance.

However, I found it nearly impossible to measure the Washington Post using WebPageTest for these reasons:

  • Measuring the Progressive Web App’s home page isn’t sufficient. Most people are reading articles which means that we need to look at differences in article loading behavior.
  • If we want to measure article pages, we have to visit multiple pages so that we can opt in to the Progressive Web App before viewing the article pages. This requires WebPageTest scripting.
  • Simply navigating to an article page doesn’t tell the true experience either. When someone selects a link in the Progressive Web App, the page content is downloaded via XHR instead of a full page load. So to replicate that experience, we need to have WebPageTest “click” the link instead of navigating to a URL.
  • We can’t use iOS emulation in WebPageTest because we need to know how iOS devices behave.
  • The WebPageTest script that sends click events doesn’t work in Safari. Not to mention the fact that real iOS devices in the WebPageTest suite are notoriously finicky.

In the end, I was forced to give up on WebPageTest. I also tried measuring page size and other behavior using Charles Proxy, but found that ad loads varied so greatly from one page load to another that I couldn’t get any meaningful data.

Some of these challenges were unique to The Washington Post and the fact that I wanted to measure iOS, but some challenges—for example, the need to script multiple pages—will be obstacles anyone wishing to measure the performance of a Progressive Web App will face.

Using video as a measuring solution

After striking out on my other attempts, I decided to record video of the page loading on my iPhone and then compare the videos. I tethered my phone to my computer and used QuickTime to record the screen.

QuickTime allows you to record your iPhone screen in a new movie recording if you tether your phone to your laptop.

I also used Charles Proxy to simulate slow networks. Charles has presets for various network speeds, but those presets seemed much too fast (16384 kbps down for 4G?) so I copied the settings from the Chrome Dev Tools throttling settings.

Charles Proxy's throttle settings set to emulate a 4G network (4000 kbps down, 3000 kbps up, 20 ms latency). I also simulated a 2G network (450 kbps down, 150 kbps up, 150 ms latency)

Then I used iMovie to combine the movies and add a timer.

iMovie's split screen feature made it relatively easy to create the comparison video.

The end result wasn’t ideal, but it was sufficient to be able to see what was happening on iOS and write about it.

Jason Grigsby

Jason Grigsby is one of the co-founders of Cloud Four, Mobile Portland and Responsive Field Day. He spends far too much time obsessing over mobile and the web. Follow him at @grigs.

Leave a Comment

Please be kind, courteous and constructive. You may use simple HTML or Markdown in your comments. All fields are required.

Let’s discuss your project! Email Us