One of the really important features that WebPagetest offers is the ability to test on various different connection types. This is critical when you're testing the end-user experience because the difference in performance between a fast Ethernet connection to a web site and a consumer connection is HUGE (like orders of magnitude huge). The main differences boil down to bandwidth (how fast your connection is - in each direction) and latency (usually referred to as the "ping times" by gamers). Bandwidth is interesting but for most sites the real killer is latency and consumer connections can easily have 50+ ms of latency before they even get to the Internet. It may not seem like much but when you add it up over tens of requests and the number of round trips each requires the effect adds up really quickly (and is the main reason it is critical to reduce the number of requests for your web site).
Most of the commercial services for testing site performance will offer you either backbone-connected test systems or (for significantly more money) testing over actual ISP connections. The backbone-connected testing is reasonably good for trending but tends to be fairly far off of the consumer experience and can easily hide changes in the number of requests or bytes that would have a large impact on the consumer experience and using real resedential lines gets to be very cost prohibitive when you're doing high-volume testing.
When we (AOL) built out our internal performance testing systems (to augment the commercial systems we use) we did a bunch of testing and analyzing and decided that a good compromise would be to use traffic-shaping technology to simulate various types of last-mile connections on our high-speed connectivity. This gives us the flexibility to test any type of consumer connection while still keeping costs under control.
There are a bunch of options for "simulating" consumer connections (proxies, browser plug-ins, etc) but it was very important to me that if we were going to simulate the connectivity that the simulation be accurate and most of the solutions fail that basic test. Unless the simulation is happening at the packet level you are going to miss a lot of the subtle behaviors that will impact the performance (TCP slow start is a great example). A bunch of them also work by proxying the browser connection which changes how the browser makes it's requests.
At a packet-level there are a bunch of different configurations from software on the same PC to software on an external PC to dedicated appliances. We started out trying a software solution but it was fairly expensive and tended to be unreliable (sometimes it would throttle and sometimes it wouldn't) so we decided a dedicated external solution would be the way to go. After a bunch of research we Narrowed down to 3 options:
Of these, dummynet was the one we liked the most. It was designed for protocol testing and could reliably simulate all of the aspects of a connection that we found interesting, was REALLY easy to configure and could scale really well (we can run several hundred test systems behind a single dummynet system, each getting a dedicated virtual pipe). Nist Net was supposed to be accurate as well but hadn't been kept up to date (didn't work with more recent kernels) and was unpleasant to configure. The netem solution is probably the most complex, requiring configuring using HTB or something else to do bandwidth and after using dummynet was excruciating to configure for a large number of systems.
This works exceptionally well for large test systems but if you're trying to deploy a small-footprint, the need for the external PC and custom networking causes problems. As we get people offering remote test systems (most recently with Daemon Solutions hosting the UK test location) it is not fair to also ask them to set up the complicated simulation configuration so I started looking around again to see if the landscape had improved any.
The first thing that grabbed my attention was that dummynet is now available on Linux. This is pretty exciting to me not so much because of the fact that it's on Linux but because Linux ALSO offers a lot of very good virtualization options (which really don't exist for FreeBSD). This opens up the possibility of having a single physical device contain the traffic shaping as well as one or more test systems (different browser versions for example).
Before you say "why don't you just run FreeBSD in a VM", I did a bunch of testing on that a while ago, trying to achieve the same goal, and the timing within a virtual environment isn't reliable enough to do good traffic shaping (it could sort of do it but the consistency just wasn't there). Running a Linux host will let the traffic shaping still happen on a physical host. It does raise the question of how consistent the test data will be for the actual browser running in a VM. I also did a bunch of testing on that (may write about it at some point because the data was not what I expected) and as long as you only use a single VM it is as reliable as a physical machine (more than one VM, even on a really high-end server tuned for it introduced enough variability to steer us away from it).
After all that I started thinking if it would be worthwhile to write my own traffic shaper for windows that worked at the packet level and did the same things we use dummynet for. I've written Windows networking drivers before and it's not what I'd call a pleasant thing to do but there are a lot of other things that it could offer - particularly packet-level details on a given test in addition to the traffic shaping. Then someone pointed me to this: http://www.akmalabs.com/downloads_netsim.php
It looks like it does most of what I'd need for a stand-alone system and would work right out of the box. I'm going to do some testing with it but if it pans out we may be able to get consistent connectivity types across the various test locations.
