Monday, October 3, 2011

Anycast and what it means for web performance


Every now and then the topic of Anycast comes up in the context of web performance so I thought I’d take a stab at explaining what it is and the benefits.

tl;dr – DNS servers should always be Anycast (and even some of the largest CDN’s are not so don’t just assume you are covered).  Anycast for the web servers/CDN is great if you can pull it off but it’s a lot less common than DNS.

Anycast – the basics

Each server on a network (like the Internet) is usually assigned an address and each address is usually assigned to a single server.  Anycast is when you assign the same address to multiple servers and use routing configurations to make sure traffic is routed to the correct server.  On private networks where there is no overlap this is pretty easy to manage (just don’t route the Anycast addresses out of the closed network).  On the public Internet things are somewhat more complicated since routes change regularly so a given machine could end up talking to different servers at different points in time as routing changes happen on the Internet (congested links, outages, and for hundreds of other reasons).

The routing behavior on a network as large as the Internet means Anycast is not a good fit for stateful long-lived connections but stateless protocols or protocols that recover well can still work.  Luckily for the web, the two foundational protocols for web traffic are largely stateless (DNS and HTTP).

DNS Anycast

By far, the most common use for Anycast on the Internet is for DNS (servers and relays).  To provide fast DNS response times for users across the globe you need to distribute your authoritative DNS servers (and users need to use DNS relays/servers close to them).

One way to distribute your servers is to give each one a unique address and just list them all as authoritative servers for your domain.  Intermediate servers running Bind 8 will try them all and favor the fastest ones but it will still use the slower ones for some percentage of traffic.  Bind 9 (last I checked anyway) changed the behavior and no longer favors the fastest so you will end up with a mix of slow and fast responses for all users.

Using Anycast you would distribute your servers globally and give them all the same IP address and you would list a single address (or a couple of Anycast addresses for redundancy) as the authoritative servers for your domain.  When a user goes to look up your domain, their DNS relay/server would always get routed to your best authoritative server (by network path, not necessarily physical geography).  Since DNS is just a request/response protocol over UDP, it really doesn’t matter if they end up talking to different physical servers for different requests.

So, as long as the routing is managed correctly, DNS Anycast is ALWAYS better than other solutions for a distributed DNS serving infrastructure (at least for performance reasons).  You should make sure that you are using Anycast DNS for moth your own records as well as any CDNs you might leverage.  It works for both the authoritative servers as well as DNS relays that users might use.  Google’s public DNS servers for end users are globally distributes but use the Anycast addresses of 8.8.8.8 and 8.8.4.4 so you will always get the fastest DNS performance regardless of where you are and what network you are on.

HTTP Anycast

Even though HTTP is not as stateless as DNS (TCP connections need to be negotiated and maintained), the connections live for a short enough time that Anycast can also work really well for HTTP – though it requires more control over the network to keep routing changes to a minimum.

Typically, geo-distribution of web servers is done by assigning them different IP addresses and then relying on geo-locating DNS to route users to the server closest to them.  It usually works well enough but there are some fairly big gotchas:

  • The geo-locating DNS server actually sees the address of the user’s DNS server, not the user themselves so it can only provide the server closest to the user’s DNS – not necessarily the user (there is a spec update to relay the actual user IP through in DNS requests so this can be done more accurately).
  • The geo-locating is only as good as the knowledge that the service has about which web servers are closest to the user’s DNS servers.  It usually works well but it’s not uncommon to see traffic routed to servers that are far away.
  • The Time To Live (TTL) on the DNS responses is usually really short (60 seconds) so that dead or overloaded servers can be pulled out as needed.  This effectively means that the DNS records can’t be cached by the user’s DNS servers and the requests all have to go back to the authoritative servers.


With Anycast, servers can be deployed globally with the same IP address.  When it works well it addresses all of the issues that using DNS to geo-locate has:

  • DNS can reply with the same IP address for all users and the address can have a long TTL and be cached by intermediate DNS resolvers.
  • In the case of a CDN, you can even assign the Anycast address directly as an A record and avoid the extra step of a CNAME lookup.
  • You don’t need to know where the user is.  Routing will take care of bringing the user to the closest server regardless of where they or their DNS server are located.
  • If you need to take a server offline, you adjust the routing so that traffic goes to the next best physical server.


I’m glossing over a LOT of the complexity in actually managing an Anycast network on the public internet but assuming you (or your provider) can pull it off, Anycast can be a huge win for HTTP performance as well.

All that said, there are only a few implementations that I am aware of for using Anycast for HTTP (and they are all CDN providers).  Anycast for HTTP should not be the main focus when picking a CDN since there are a lot of other important factors – the most important of which is to make sure they actually have edge nodes near your users (if you have a lot of users in Australia then pick a provider with edge nodes in Australia FIRST, then compare other features).

17 comments:

  1. DNS services using Anycast for TCP is fairly common (at least for UltraDNS it is), particularly because the new DNSSEC packets are bigger - and a lot can push over 512K when you add the RRSIGs, etc. UltraDNS also runs a redirection / URL shortening type service over HTTP that uses Anycast as well.

    ReplyDelete
  2. I did some analysis on the DNS based Global Load Balancing used by many CDNs and how accurate / confident they are in their answers. This is the most popular alternative to Anycast for HTTP traffic. Plan to follow it up with some Anycast comparisons. - bit.ly/oKRkVB

    ReplyDelete
  3. Oh and at Highwinds we use Anycast for HTTP traffic as well.

    ReplyDelete
  4. @Rich, great data, thanks for sharing. It looks a lot like what I'm used to seeing with the spotty accuracy of DNS-based gslb. I didn't realize Highwinds also did anycast, updating the post.

    ReplyDelete
  5. We need a low(or atleast reasonable) cost service provider for Anycast! Currently it seems only the huge ISP level people can pull it off, not the average joe.. I look forward to the day when Amazon or someone offers something like have ec2 instances in different regions accessible by a single IP!... ok I need to stop dreaming and get back to work...

    ReplyDelete
  6. For CDN: Cotendo & Edgecast are anycast
    For DNS: DYN, Verisign, Cotendo are anycast
    it makes a huge difference! when you have a CDN that relies on 10 cnames it's a nightmare

    ReplyDelete
  7. @Sajal It's called CloudFlare, and it's free.

    ReplyDelete
  8. DNSMadeEasy.com is Anycast DNS as well and a lot cheaper than Dyn/UltraDNS.

    We're using Edgecast as our CDN, but is setup through a CNAME, not Anycast. I'm not sure if anycast was ever offered to us when we signed up.

    ReplyDelete
  9. @Noodles, Anycast and CNAME are not mutually exclusive. You can CNAME to a FQDN that resolves to an anycast IP (that's how most do it actually).

    The distinguishing feature of anycast is that it resolves to the same IP address no matter where you query from globally.

    ReplyDelete
  10. A few corrections from what I have seen in my own research.
    First Cotendo does NOT use ANYCAST for HTTP, rather a collection of unicast IPs managed vie global load balancing DNS. As someone else pointed out Cloudflare does use HTTP ANYCAST. Finally Edgecast is a bit of a hybrid. They use DNS load balancing to direct users to one of 3 Anycast IPs that represent different regions of the world. Many people that use ANYCAST on a global scale do this as edge selection is done by hop counts and depending on how you broadcast your space in different parts of the world (are you behind another an additional AS in some PoPs but not in others) can make for undesirable hop count selections (and cause users to go to PoPs you might not want them to go to).

    ReplyDelete
    Replies
    1. what you're saying is not true released today Cotendo It is a subsidiary of Akamai the largest content delivery networks and any cast DNS servers in the world. If what you said about Cotendo work true it would be a multicast geo-DNS not unicast. Regardless check their website it's an IP anycast.

      The best DNS is by far hosted and served around the world by DynECT they are truly a remarkable and by far in life and in the best.
      All any cast networks have the ability to load balanced DNS edge cast is possibly hybrid but all hybrid is is in a test system that relies on a unicast system for uptime when needed. It's called IP Anycast today.

      The idea that their are 3 IP addresses for the entire edgecast CDN/DNS system that make up different regions is false. While yes of course they do make up different regions called zones in DNS their are far more than simply 3 IP addresses and far more than simply 3 zones. I'm not trying to be difficult or anything like that I'm simply stating the case as it is.

      Geo DNS preferably any cast is by far the fastest @noodes stated the distinguishing feature of any cast is it uses the same IP address no matter where it goes. The same goes for unicast and the same goes for multicast DNS so I don't really think that is correct.

      Delete
    2. @Anynomous - Things have changed a fair bit over the last 2 years (including Contendo's acquisition) and I'd be hesitant to assume that any single DNS vendor is "by far" the best - there are a good number of excellent ones out there and when you're up at the price points of some of the bigger ones ($200/mo starting for DynECT right now for example) odds are that you are going to be doing a deep dive into their full architecture.

      I don't understand your last paragraph very well but unicast does not work the same way as anycast. In unicast, each DNS end point has a dedicated IP address and requests to it from anywhere in the globe will end up at the same server (or load balancer) while with anycast they all share an address and network routing takes you to the instance that is "closest" to you automatically: http://dyn.com/blog/unicast-vs-anycast-dns-nameserver-routing/ - The Google public DNS servers are a good example of this where 8.8.8.8 is a shared address that gets routed to dozens of different geographies depending on which server you are closest to.

      Delete
  11. Thanks @Rich. I just pulled the explicit list from the article since it was bouncing around (and subject to change in the future).

    ReplyDelete
  12. sorry as I know this is a very later comment but I am curious on http anycast.. So can anyone elaborate on how you handle a routing event midstream of an http request that cause the routing path to change, sending the request to another location.

    Since the next web server or LB will have know knowledge of the SEQ# ( and its not SYN), there will be a RST. So how do you handle this ?

    ReplyDelete
  13. Cachefly had a great presentation at nanog a while ack about it: http://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf

    The key is to keep the routs stable so that any specific client is usually sticky to the same server. In their measurements, routing changes happen infrequently enough that they don't really impact short-lived HTTP connections any more than the regular noise levels.

    ReplyDelete
  14. okay let me ask another question. So if I have services behind a load balance appliance(SLB) (VIPS) how would the route withdraw work in this scenario?

    I advertise a /24 from multiple locations and then have a handful of VIPS in each location for different services that are the same in each location. If one of these VIPS becomes unavailable, that entire /24 would be withdrawn from BGP prefix being advertised. So I have stopped all routing to that location for the VIPS in the anycast /24.... How can you have multiple services/VIPs and geet around this?


    thanks for your previous response...

    ReplyDelete
  15. You would not want to use it to manage individual services (which is why it works well for a CDN where they take an entire region down if there is a sufficiently large enough problem to justify it). Application-specific failover should be handled behind the VIPS.

    ReplyDelete

All comments are moderated and may take a while to appear.