Testing for Frontend SPOF

Steve Souders had a great blog post last year that talked about Frontend Single Points Of Failure (SPOF).  Given the continuing rise in 3rd-party widgets on pages it is becoming increasingly important and I realized that there weren't any good tools for testing for it.  Seemed like the perfect opportunity to piece something together so that's exactly what I did.

Testing for Frontend SPOF

Probably the most critical part of testing a failure of a 3rd-party widget is to make sure you get the failure mode correct.  When these things fail, the servers usually become unreachable and requests time out.  It is important to replicate that behavior and not have the requests fail quickly, otherwise you will see what your site looks like without the content but the experience won't be right (the real experience is Sooooooo much worse).

I looked around for a well-known public blackhole server but couldn't find one so I went ahead and set one up (feel free to use it for your testing as well):

blackhole.webpagetest.org (aka 72.66.115.13)

A blackhole server is a server that can be routed to but all traffic gets dropped on the floor so it behaves exactly like we want when testing the failure mode for 3rd-party widgets.

With the blackhole server up and running you can now use it for testing manually or through tools like Webpagetest.

Browsing the broken web

For the purposes of this example I'll be "breaking" the twitter, Facebook and Google buttons as well as the Google API server (jquery, etc) and Google Analytics.

Now that we have a blackhole server, breaking the web is just a matter of populating some entries in your hosts file (C:\Windows\System32\drivers\etc\hosts on windows).  Go ahead and add these entries and save the updated hosts file:

72.66.115.13 ajax.googleapis.com     

72.66.115.13 apis.google.com         

72.66.115.13 www.google-analytics.com

72.66.115.13 connect.facebook.net    

72.66.115.13 platform.twitter.com    

...and go browse the web.  It shouldn't take you long to find a site that is infuriatingly painful to browse.  Congratulations, you just experienced a Frontend SPOF - now go fix it so your users don't have to feel the same pain (assuming it is a site you control, otherwise just yell at the owner).

Testing it with WebPagetest

It's a lot easier to discover broken content just by browsing using the hosts file method, but if you find something and need to make the case to someone to get it fixed, nothing works better than a WebPagetest video.

First, test the site as you normally would but make sure to check the "capture video" option (and it's probably not a bad idea to also give it a friendly label).

Next, to capture the broken version of the site you will need to use a script (largely just copy and paste).  You need to send the broken domains to the blackhole and then visit the page you are trying to test:

setDnsName ajax.googleapis.com blackhole.webpagetest.org

setDnsName apis.google.com blackhole.webpagetest.org

setDnsName www.google-analytics.com blackhole.webpagetest.org

setDnsName connect.facebook.net blackhole.webpagetest.org

setDnsName platform.twitter.com blackhole.webpagetest.org

navigate your.url.com

Just paste the script into the script box (with the correct URL to be tested), make sure capture video is checked and that you have a friendly label on the test.

Finally, go look at the test history, select the tests that you ran and click compare (the history works best if you log into the site before submitting your tests).

And what would be the fun in it without an example.  Here is what happens to Business Insider when Twitter goes down (yeah, THAT never happens): http://www.webpagetest.org/video/view.php?id=111011_4e0708d3caa23b21a798cc01d0fdb7882a735a7d

Yeah, so it's normally pretty slow but when Twitter goes down the user stares at a blank white screen for 20 seconds!  At that point, Business Insider itself may as well be down.  Luckily it can easily be solved just by loading the twitter button asynchronously.

Anycast and what it means for web performance

Every now and then the topic of Anycast comes up in the context of web performance so I thought I’d take a stab at explaining what it is and the benefits.

tl;dr – DNS servers should always be Anycast (and even some of the largest CDN’s are not so don’t just assume you are covered).  Anycast for the web servers/CDN is great if you can pull it off but it’s a lot less common than DNS.

Anycast – the basics

Each server on a network (like the Internet) is usually assigned an address and each address is usually assigned to a single server.  Anycast is when you assign the same address to multiple servers and use routing configurations to make sure traffic is routed to the correct server.  On private networks where there is no overlap this is pretty easy to manage (just don’t route the Anycast addresses out of the closed network).  On the public Internet things are somewhat more complicated since routes change regularly so a given machine could end up talking to different servers at different points in time as routing changes happen on the Internet (congested links, outages, and for hundreds of other reasons).

The routing behavior on a network as large as the Internet means Anycast is not a good fit for stateful long-lived connections but stateless protocols or protocols that recover well can still work.  Luckily for the web, the two foundational protocols for web traffic are largely stateless (DNS and HTTP).

DNS Anycast

By far, the most common use for Anycast on the Internet is for DNS (servers and relays).  To provide fast DNS response times for users across the globe you need to distribute your authoritative DNS servers (and users need to use DNS relays/servers close to them).

One way to distribute your servers is to give each one a unique address and just list them all as authoritative servers for your domain.  Intermediate servers running Bind 8 will try them all and favor the fastest ones but it will still use the slower ones for some percentage of traffic.  Bind 9 (last I checked anyway) changed the behavior and no longer favors the fastest so you will end up with a mix of slow and fast responses for all users.

Using Anycast you would distribute your servers globally and give them all the same IP address and you would list a single address (or a couple of Anycast addresses for redundancy) as the authoritative servers for your domain.  When a user goes to look up your domain, their DNS relay/server would always get routed to your best authoritative server (by network path, not necessarily physical geography).  Since DNS is just a request/response protocol over UDP, it really doesn’t matter if they end up talking to different physical servers for different requests.

So, as long as the routing is managed correctly, DNS Anycast is ALWAYS better than other solutions for a distributed DNS serving infrastructure (at least for performance reasons).  You should make sure that you are using Anycast DNS for moth your own records as well as any CDNs you might leverage.  It works for both the authoritative servers as well as DNS relays that users might use.  Google’s public DNS servers for end users are globally distributes but use the Anycast addresses of 8.8.8.8 and 8.8.4.4 so you will always get the fastest DNS performance regardless of where you are and what network you are on.

HTTP Anycast

Even though HTTP is not as stateless as DNS (TCP connections need to be negotiated and maintained), the connections live for a short enough time that Anycast can also work really well for HTTP – though it requires more control over the network to keep routing changes to a minimum.

Typically, geo-distribution of web servers is done by assigning them different IP addresses and then relying on geo-locating DNS to route users to the server closest to them.  It usually works well enough but there are some fairly big gotchas:

• The geo-locating DNS server actually sees the address of the user’s DNS server, not the user themselves so it can only provide the server closest to the user’s DNS – not necessarily the user (there is a spec update to relay the actual user IP through in DNS requests so this can be done more accurately).
• The geo-locating is only as good as the knowledge that the service has about which web servers are closest to the user’s DNS servers.  It usually works well but it’s not uncommon to see traffic routed to servers that are far away.
• The Time To Live (TTL) on the DNS responses is usually really short (60 seconds) so that dead or overloaded servers can be pulled out as needed.  This effectively means that the DNS records can’t be cached by the user’s DNS servers and the requests all have to go back to the authoritative servers.

With Anycast, servers can be deployed globally with the same IP address.  When it works well it addresses all of the issues that using DNS to geo-locate has:

• DNS can reply with the same IP address for all users and the address can have a long TTL and be cached by intermediate DNS resolvers.
• In the case of a CDN, you can even assign the Anycast address directly as an A record and avoid the extra step of a CNAME lookup.
• You don’t need to know where the user is.  Routing will take care of bringing the user to the closest server regardless of where they or their DNS server are located.
• If you need to take a server offline, you adjust the routing so that traffic goes to the next best physical server.

I’m glossing over a LOT of the complexity in actually managing an Anycast network on the public internet but assuming you (or your provider) can pull it off, Anycast can be a huge win for HTTP performance as well.

All that said, there are only a few implementations that I am aware of for using Anycast for HTTP (and they are all CDN providers).  Anycast for HTTP should not be the main focus when picking a CDN since there are a lot of other important factors – the most important of which is to make sure they actually have edge nodes near your users (if you have a lot of users in Australia then pick a provider with edge nodes in Australia FIRST, then compare other features).