Results 1 to 32 of 32
  1. #1
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331

    Help Understanding Anycast

    I operate a few DNS servers across multiple networks. One at Amazon, another at a secondary datacenter, another server in another location, etc.

    Right now I am using unicast. I am really hoping to make the transition to anycast for grater up-time, lower latency, and the ability to easily add more physical servers into the cloud as needed.

    Can someone help me get started in the right direction?

    1.) Can I anycast across totally different networks operated by different companies?

    2.) What is the best way to get started? Is there a network we can go to that can help us set this up?

    In the ideal world I would like to be assigned a static IP address, and have someone operating a BGP router host this for me. But from reading it does not sound that simple..? Would love a little direction.

  2. #2
    Join Date
    Apr 2010
    Posts
    491
    Quote Originally Posted by Rebies View Post
    I operate a few DNS servers across multiple networks. One at Amazon, another at a secondary datacenter, another server in another location, etc.

    Right now I am using unicast. I am really hoping to make the transition to anycast for grater up-time, lower latency, and the ability to easily add more physical servers into the cloud as needed.

    Can someone help me get started in the right direction?

    1.) Can I anycast across totally different networks operated by different companies?

    2.) What is the best way to get started? Is there a network we can go to that can help us set this up?

    In the ideal world I would like to be assigned a static IP address, and have someone operating a BGP router host this for me. But from reading it does not sound that simple..? Would love a little direction.
    1 Yes in a fashion you need to have a BGP session with each of them and at least a /24 to dedicate to anycast.

    2 There are multiple providers than can do anycast for you. Using multiples of them is still far cheaper than setting it up yourself unless your fielding truly massive amounts of DNS or playing games with tailored responses.

    Getting a /22 (minimum allocation)out of ARIN will be tough and some of your providers may not support a BGP routed sessions. Doing anycast well requires a lot more than a handful of locations.

    DNSSEC does not really work well with anycast (yes unicast secondary can pick up the slack after timeouts). So do you really want to go down a dead end road?

  3. #3
    Join Date
    Jan 2010
    Posts
    652
    0. http://en.wikipedia.org/wiki/Anycast
    1. yes
    2. http://en.wikipedia.org/wiki/Anycast and http://www.dnsmadeeasy.com/

    Seriously, you need to ask yourself *WHY* you want to do your own anycast if you do not already have BGP setup in multiple locations.
    Then, ask yourself WHY you want to do it yourself when companies like DNSMadeEasy are so cheap.

  4. #4
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    Yes it is our own, custom written DNS software so we can not go with a DNS provider. And we are fielding a large amount of DNS traffic as well. You bring up a good point of DNSSEC that I had not thought about. We are trying to make our DNS more scaleable, so using a solution like DNSMadeEasy just does not work for us.

  5. #5
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    I highly doubt Amazon would let you announce BGP to them, but I could be wrong. You're going to need your own ASN and either your own /24 or a portable /24 from one of your providers.

    If you don't already have this in place, it would probably be advisable to work with a provider that has a /24 already dedicated to anycast, and can provide you /32's within it for your uses. They'll just either need to have their own backbone, or upstream providers that will allow them to announce /32's across their backbones. That way you can still manage your own servers and applications, but have the actual BGP and anycast part taken care of for you. You'll still need to setup your own instance of quagga, bird, openbgpd, or such on your DNS servers to advertise the /32's to the provider though.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  6. #6
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    hhw - that is very helpful! let me understand one thing. You are saying if I work with someone who already has a /24 dedicated to anycast and gives me a few /32's to use.. They can route them where I want. But you are doubting that Amazon would allow us to announce BGP, so hosting some instances at Amazon would (by your guess) not work? Or did I misintrepret and with someone who has a /24 anycast, they could still point to Amazon servers? The reason I'm interested in Amazon is the extremely easy ability to spin up new instances on demand in multiple locations.

  7. #7
    Join Date
    Feb 2011
    Posts
    580
    You don't sound to be ready for your own anycast setup. To do it you need not only IP space (minimum of /24 dedicated to anycast) but also multiple locations with routers. Also for anycast to actually improve latency you need to use the same tier 1 providers in all locations. Otherwise your traffic will not take lowest latency routes. If you use Amazon you are on the opposite end of the infrastructure control spectrum of where you need to be in order to operate anycast. If you want to do it right you need first year budget of over $100k to get benefits from your own anycast network.

  8. #8
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by Rebies View Post
    hhw - that is very helpful! let me understand one thing. You are saying if I work with someone who already has a /24 dedicated to anycast and gives me a few /32's to use.. They can route them where I want.
    Not where you want, they will be announcing that /24 in whatever locations they're announcing that /24. Out of those locations, you can have a dedicated server, vps, or what have you and announce the /32 to the provider to then propagate to their upstreams.


    Quote Originally Posted by Rebies View Post
    But you are doubting that Amazon would allow us to announce BGP, so hosting some instances at Amazon would (by your guess) not work? Or did I misintrepret and with someone who has a /24 anycast, they could still point to Amazon servers? The reason I'm interested in Amazon is the extremely easy ability to spin up new instances on demand in multiple locations.
    Correct, you would not be able to use Amazon cloud instances in this scenario, or any scenario with anycast really. If you were pointing to Amazon with DNS, that would defeat the purpose of running anycast DNS in the first place.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  9. #9
    Join Date
    Oct 2007
    Posts
    446
    Quote Originally Posted by Rebies View Post
    Yes it is our own, custom written DNS software so we can not go with a DNS provider. And we are fielding a large amount of DNS traffic as well. You bring up a good point of DNSSEC that I had not thought about. We are trying to make our DNS more scaleable, so using a solution like DNSMadeEasy just does not work for us.
    It sounds like you should use some of the resources on custom writing your own DNS software and invest in some network skills on your staff. Many of your statements do not belong together.

    In regards to hosting in Amazon and wanting to run IP Anycast in their environment. This makes zero technical sense.
    This is would be similar to walking into a Burger King and asking the cashier to buy a car. Amazon does not even offer the services you are requiring... not even close.



    Quote Originally Posted by Rebies View Post
    The reason I'm interested in Amazon is the extremely easy ability to spin up new instances on demand in multiple locations.
    Yeah... I like shared hosting too. Especially at GoDaddy because it is free. It doesn't mean that I can run Anycast services over it.



    At $0.20 per million queries (which is what DNS Made Easy charges the high bulk queries) it would be very tough to make a valid argument to not outsource DNS.

  10. #10
    Join Date
    Aug 2003
    Location
    127.0.0.1
    Posts
    273
    Quote Originally Posted by DMDM View Post
    If you want to do it right you need first year budget of over $100k to get benefits from your own anycast network.
    That's not really the case any longer - if you're doing anycast in the cloud, although it certainly won't be inexpensive depending on your yard stick.

    @OP you can use a service like the already mentioned & excellent DNSMadeEasy - the guys over there really know what they're doing - and easily with your existing infrastructure. Just slave your zones to their service - you can do this automatically via several different methods.

    Beyond that Anycast has quite a few considerations on the software side when it comes to design that need to be considered. DNS may be one of the easier services to anycast at a basic level, but once you consider the many aspects of managing data/services in multiple locations it can get complex.

    Unless you have a compelling need, the expertise, or the budget the kind of drop in solution someone like DNSMadeEasy offers may make the most sense.
    Getting Cloudy in the US, Europe, & South Asia.

  11. #11
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    Quote Originally Posted by hostvirtual View Post
    Unless you have a compelling need, the expertise, or the budget.
    The need is there. Our software is custom, and we can't use typical DNS software. We have too frequent of changes in the zones, and the responsiveness was not there with off-the-shelf software. As a domain registrar the DNS software is too important and critical to us (and custom) to outsource.

    Expertise: I wish we had this with global networks. Right now we don't.

    Budget: I am considering this as I find I am getting into an expensive and global network issue I did not know was going to be required for Anycast. But I know it would be a huge benefit to our company.

    I absolutely appreciate the opinions and comments. I would love to hear more. I'm still a deer in headlights just a little bit. Sorry I am not an expert with BGP routing.

  12. #12
    Join Date
    Apr 2010
    Posts
    491
    Quote Originally Posted by Rebies View Post
    The need is there. Our software is custom, and we can't use typical DNS software. We have too frequent of changes in the zones, and the responsiveness was not there with off-the-shelf software. As a domain registrar the DNS software is too important and critical to us (and custom) to outsource.

    Expertise: I wish we had this with global networks. Right now we don't.

    Budget: I am considering this as I find I am getting into an expensive and global network issue I did not know was going to be required for Anycast. But I know it would be a huge benefit to our company.

    I absolutely appreciate the opinions and comments. I would love to hear more. I'm still a deer in headlights just a little bit. Sorry I am not an expert with BGP routing.
    Pretty much it boils down to you needing at least a VPS (more depending on what your software does) a /24 you can use and and BGP ASN. You need a LOT of points to make this work well and they need to be the correct points. If you just need fast updates dnsmadeeasy can do that if you need to tailor the response per query. It's been a long time since I set one up but expect to need at least 40 sites to be effective, with peering cross connects often costing far more than the servers.

  13. #13
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by Rebies View Post
    Budget: I am considering this as I find I am getting into an expensive and global network issue I did not know was going to be required for Anycast. But I know it would be a huge benefit to our company.
    The expense is dependent on the number of locations you want BGP anycast out of. Two locations isn't going to cost that much. Twenty will cost quite a lot. You can try to reduce your cost as much as possible per location, but your overall costs will still scale up linearly with each additional location. You don't need to have anycast in every possible location, but if the ultimate goal is low latency, you'll want a good geographical spread. At least 6 in North America for Northwest, Southwest, North Central, South Central, Northeast, and Southeast. Europe can be reasonably well covered by one location as it's geographically small. Asia is a large, spreadout region as well but due to expensive and limited connectivity may or may not be worth your while. Oceania is somewhat the same, just smaller.

    As I've mentioned previously, you don't need your own ASN and /24 if you can leverage off someone else's BGP anycast, and just advertise /32's. It's still not going to be cheap, as very few providers would offer such a service, and those that do have their own significant costs to recoup as well as needing to charge enough to make it worth their while to offer it. You're also looking at using premium carriers, as geographical coverage won't help as much with latency if your routing isn't spot on. You should be looking at 4 digits to have a basic, reasonable spread of locations and easily into 5 digits and beyond if you want comprehensive coverage around the world. If you're not operating at that scale, just stick to geoIP based results, which can still offer some latency benefits, just with much less accuracy.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  14. #14
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    Interesting you bring up geoIP based rules, that was something I was brainstorming last night. Our nameservers are served based on geoIP, and then the local servers would respond with the actual requests. Not as great, but it could work a little bit.

    When you say 4 digits to have a basic spread, and easily into 5 digits with comprehensive coverage... Are you talking monthly or yearly on that? $10,000 for yearly to leverage someone else's network would defiantly be within our budget I would guess.

    Last night I sent a few contact requests off to companies that have these types of networks. vr.org, internap come to mind. Hopefully one of these 5 or 6 companies I contacted would be able to help.

  15. #15
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    Update... HostVirtual is the first company to come back to me with options. They have a handful of options, if using our own IP space it would be as little as $5k to $15k a year, depending on how many locations we were to use. Anyone else have suggestions on companies that have the backbone and setup to allow us to do this? I have not heard back from a lot of companies, but I expect to early next week.

  16. #16
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by Rebies View Post
    When you say 4 digits to have a basic spread, and easily into 5 digits with comprehensive coverage... Are you talking monthly or yearly on that? $10,000 for yearly to leverage someone else's network would defiantly be within our budget I would guess.
    Monthly. The range can be quite varied per location, but I can't see anyone offering it for less than $100/mo, even off VPS, but most likely multiple times that. For 10 locations, that would easily put you at 4 figures a month. For comprehensive coverage, some of the more obscure locations could get up to $1,000/mo just on their own.

    Quote Originally Posted by Rebies View Post
    Update... HostVirtual is the first company to come back to me with options. They have a handful of options, if using our own IP space it would be as little as $5k to $15k a year, depending on how many locations we were to use.
    They're using dnsmadeeasy for their own DNS, so I'm not sure how experienced they are with anycast if they don't run it for their own use. That's a moot point though if you have to use your own IP space, which you don't have, do you? If you did, that would open up a lot more options as the provider would then only need to support regular BGP, and not really have to do anything anycast specific other than have symmetric transit providers.

    Asymmetric transits can lead to problems once it reaches that transit provider, as it will end up preferring locations where they're used as opposed to the nearest geographical location. All the transit providers used would need to offer BGP community strings that would prevent advertisements from being redistributed to other locations in that case, and it would be non-trivial to have all the necessary BGP policies in place to make use of those strings.
    Last edited by hhw; 11-16-2013 at 08:27 PM.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  17. #17
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    No, right now we don't have our own IP space, just rented from our upstreams. But I would not be opposed to going down that road of obtaining our own IP space so that we can do this.

  18. #18
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by Rebies View Post
    No, right now we don't have our own IP space, just rented from our upstreams. But I would not be opposed to going down that road of obtaining our own IP space so that we can do this.
    Keep in mind that in doing so, you'll be taking on all the BGP configuration on yourself as well. That will likely involve hiring a network engineer to architect the design, and be available to you on an ongoing basis as needed. You'll want to factor that into your TCO considerations as well. Be sure to find one specifically with backbone level experience, as campus and data centre environments generally will not involve the same type of network architectures as what would be needed for any anycast setup.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  19. #19
    Join Date
    Aug 2003
    Location
    127.0.0.1
    Posts
    273
    Quote Originally Posted by hhw View Post
    They're using dnsmadeeasy for their own DNS, so I'm not sure how experienced they are with anycast if they don't run it for their own use.
    We know and like the guys at Tiggee very well. Having experience with Anycast, and wanting to run (our own) DNS are two very different things.
    Getting Cloudy in the US, Europe, & South Asia.

  20. #20
    Join Date
    Jan 2010
    Posts
    652
    Not sure how you jumped from asking basic questions to getting quotes without listing your requirements.

    You need to define your requirements - how many zones, how much traffic, how reliable, etc.

    On the low end, you could go with 5 zones: Asia, Australia, West US, East US, Europe.
    Then go with 10 vps, 2 per zone. Run your specialized DNS servers on those vps. If traffic picks up in one zone, increase CPU on that vps, or go to a dedicated server.
    Then use dnsmadeeasy global to point users to your DNS server IPs.
    Redundant, reliable, robust, cheap.

  21. #21
    Join Date
    Aug 2002
    Location
    Denver, CO
    Posts
    331
    Sorry Techee.. I am trying to learn what is out there and possible as I go. This thread is tremendously helpful and I'm in no way settling on what solution we will go with yet. But as you can imagine, for someone like me who does not do the nit-and-gritty of networking myself, there is a bit of learning.

  22. #22
    Join Date
    Mar 2012
    Posts
    52
    Well, I see so many people trying to operate their own anycast (some of them are contacting me, since I operate one with about 30 locations as well) until they notice how time and money consuming this process can be... you will notice that most of the bgp communities provided by transits are far away from working... being able to make sure that you have the same transits in all your locations can be hard (prepending is not a real solution, but meh often the only you have)
    You should ask yourself if it is *really* required to have your own anycast and what kind of benefits you are hoping to have.

    However, I can really vote for HostVirtual - if someone knows what he's doing then it's Mark. He helped me with various things and is always up for interesting talks.

    But yes, it actually is possible to get a smaller net (like /29 or even /32) from an existing anycast network to any servers you want to have. We're doing the same (iBGP) for ensuring redundancy within our network. It's really cool what you can do with BGP.

    Quote Originally Posted by hhw View Post
    They're using dnsmadeeasy for their own DNS, so I'm not sure how experienced they are with anycast if they don't run it for their own use. That's a moot point though if you have to use your own IP space, which you don't have, do you? If you did, that would open up a lot more options as the provider would then only need to support regular BGP, and not really have to do anything anycast specific other than have symmetric transit providers.
    That actually doesn't mean anything, since this could be done by them to have their main website still up even if there is any kind of failure on their network to inform customers, etc.

    It's similar to why a Web Host should NOT use their own services for their website (or if they do, then they should at least have a backup on another network). This is because if their network or their servers go down then the customers would not have the easiest direct line of contact with the provider. To prevent this you distribute some of your more important files around to other hosts. This is called Risk Management and you should totally read into it (it's quite useful!). While yes some people may want the owner to use his own services to present himself, it's bad in terms of Risk Management because they really are putting all of their eggs in one basket.

  23. #23
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by Amfy View Post
    That actually doesn't mean anything, since this could be done by them to have their main website still up even if there is any kind of failure on their network to inform customers, etc.
    That may be applicable when you're running out of a single location or transit provider, but if you have multiple locations around the world and are multi-homed in each of them, it makes a lot less sense to run your site on another network for redundancy purposes. And considering the site is in fact on their network, that's clearly not the case here anyway.

    Is your anycast solution hosted in part with them? Is your need for prepending a result of working with them, or with other providers?

    Quote Originally Posted by hostvirtual View Post
    We know and like the guys at Tiggee very well. Having experience with Anycast, and wanting to run (our own) DNS are two very different things.
    Seems like a potential missed opportunity to not leverage your own architecture, but I can certainly understand wanting to stay focused and not be distracted by anything other than your core product. As you're implying that you are experienced with anycast, can you elaborate on what that experience is?

    You appear to be using 11 different transits, but clearly not all 11 of them are at all 15 of your locations, as some of them appear to be local data centre networks in some of the cities where you host. Do all of these local networks you use provide provisions to prevent redistributing your announcements outside of their region? We're always thinking about Asia, but haven't found the justification for the higher costs as of yet, and the difficulty in having the same transit providers in place. On the other hand, if working with regional transit providers, it seems rather unlikely that those announcements will end up being preferred outside that region. If you have a good solution in place, we might be interested in working with you there and in other areas outside of North America and Europe, as we'd like to expand our presence but don't have enough traffic in those regions to justify our own infrastructure there as of yet.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  24. #24
    Join Date
    Aug 2003
    Location
    127.0.0.1
    Posts
    273
    Seems like a potential missed opportunity to not leverage your own architecture,
    FWIW - The excellent DNSMadeEasy.com itself does not appear to be Anycast, but its services are - just as an example since you've mentioned them.

    I don't know you although we've both been on WHT about the same amount of time. I couldn't help notice that you used to work for Peer1 on their anycast/cdn services.

    They now appear to use Cloudflare (a great service) to power their site vs. their own RapidEdge(tm) product. I'm sure there are various reasons, business and otherwise for this

    We do happen to anycast our site, but only in some circumstances.

    Early on in this thread I'd mentioned to the @OP how complex Anycast can be, and recommended he consider using DNSMadeEasy to slave his zones as a way to get the benefit of their Anycast platform to leverage their DNS infrastructure for a quick win.

    Beyond business goals, evaluating the service or application and understanding how it will react when Anycast is a critical component. It's very common to get requests to anycast an interactive/live service such as voip or a real time game for example, which can have significant impact on the technical design of the software being deployed.

    You appear to be using 11 different transits, but clearly not all 11 of them are at all 15 of your locations
    We're an IaaS provider with multiple services. Our sole (or even primary) purpose isn't Anycast, but we do operate with sets of common transit in different geographical locations. We're a building block.

    We're always thinking about Asia, but haven't found the justification for the higher costs as of yet, and the difficulty in having the same transit providers in place.
    Asia is a very interesting place, and it would be impossible to find the same transit providers globally that have local reach into specific regions. Being able to control upstream announcements with communities (or our announcements upstream) are as you mention two of the critical components in successfully deploying an Anycast platform.

    Beyond just setup - ongoing care and feeding to constantly adjust for different conditions is certainly required. Business goals have to also consider that there is no 100% - in the end if someone has multiple paths to you, and wants to force traffic over a certain route there is little, if anything you as an operator can do.

    I often use an analogy - say you've given directions to your home for an event. If a subset of people you've invited decide to take the scenic route that's 5 hours longer, even after you're provided them with maps and markers it's out of your hands.

    we might be interested in working with you
    Always happy to make new friends / explore possible synergies. I've pinged you offline with my details. It looks like we're both on the west coast
    Getting Cloudy in the US, Europe, & South Asia.

  25. #25
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by hostvirtual View Post
    I don't know you although we've both been on WHT about the same amount of time. I couldn't help notice that you used to work for Peer1 on their anycast/cdn services.

    They now appear to use Cloudflare (a great service) to power their site vs. their own RapidEdge(tm) product. I'm sure there are various reasons, business and otherwise for this
    PEER1 was pretty early into the anycast game, using it both for CDN, and the first to use it for DDoS mitigation. Both started out as skunk works projects out of the Vancouver office, and were unfortunately never fully solidified as products despite pretty good traction. I could go on and on about the reasons why, but I'll just leave it at this: the company made a conscious move away from doing any R&D internally, preferring to focus on customer service and using external best-of-breed solutions instead of developing in house. For these and various other reasons, all the developers in Vancouver eventually moved on, with most of us ending up at Astute. We're still leveraging PEER1's network to do our anycast, and they've been great to work with on that front, but don't have any presence outside North America or Europe. We do miss having our own backbone immediately at our disposal though, as visibility and control were obviously much more convenient.

    Quote Originally Posted by hostvirtual View Post
    It's very common to get requests to anycast an interactive/live service such as voip or a real time game for example, which can have significant impact on the technical design of the software being deployed.
    Interesting uses for anycast. Considering that these would be even more dependent on keeping state than HTTP(S), I'm curious how these hold up to routing changes. I still lean towards the more conservative approach of using anycast for stateless applications like DNS only. The DNS specific problems like non-local lookup servers are in my opinion easier to deal with than synchronizing state between different locations for seamless failover during routing changes. We're probably going to go the way of generating custom geoIP data using in-line flows analysis and overriding responses when needed. I'm always interested to hear about others' experiences with full anycast though.

    Quote Originally Posted by hostvirtual View Post
    Asia is a very interesting place, and it would be impossible to find the same transit providers globally that have local reach into specific regions. Being able to control upstream announcements with communities (or our announcements upstream) are as you mention two of the critical components in successfully deploying an Anycast platform.

    Beyond just setup - ongoing care and feeding to constantly adjust for different conditions is certainly required. Business goals have to also consider that there is no 100% - in the end if someone has multiple paths to you, and wants to force traffic over a certain route there is little, if anything you as an operator can do.
    Do all the networks you work with in Asia support such community strings, even the local/regional ones? The problems I have with prepending is:
    1) localpref still takes precedence
    2) can't be implemented with enough precision
    As such, I would much prefer to restrict announcements.

    Quote Originally Posted by hostvirtual View Post
    Always happy to make new friends / explore possible synergies. I've pinged you offline with my details. It looks like we're both on the west coast
    Likewise Let's continue any conversation that might be relevant to others in the community here, but I'll contact you off the forums for anything specific to us.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  26. #26
    Join Date
    Feb 2011
    Posts
    580
    Quote Originally Posted by hhw View Post
    We're still leveraging PEER1's network to do our anycast, and they've been great to work with on that front, but don't have any presence outside North America or Europe.
    Peer1 is not great for anycast by any stretch of imagination simply because they are a Tier 2 network. Same applies to bigger networks like nlayer. There are couple reasons for that:

    1. They have assymetric uplinks at their locations. For example look at their Vancouver routing for Level3 customers and compare that to Toronto...

    2. As Tier 2 networks they have transit uplinks. This means that their transit providers setup their routers to prefer the routes of their customers irrespective of any other parameters.

    The only way to properly balance anycast is to mix Tier 1 providers at each location. Ideally you would use Tier 1 providers that do not ignore the routing policy attributes that your routers export.

  27. #27
    Join Date
    Oct 2007
    Posts
    446
    Quote Originally Posted by hhw View Post
    They're using dnsmadeeasy for their own DNS, so I'm not sure how experienced they are with anycast if they don't run it for their own use.
    HostVirtual actually have their own routers and virtual instances so there is no reason why they could not offer their own IP Anycast services / solution. I think they even have IPv6, so you could get that anycasted as well.

    Just because a company uses DNS Made Easy doesn't mean they don't know how to set up a network that uses IP Anycast. They might just not need to worry about DNS. You have to understand that DNS Made Easy specializes in DNS and is optimized in speed for their clients. This is all they do.
    http://solvedns.com/dns-comparison/2013/10

    HostVirtual offers a different service. But they could probably offer an IP Anycast solution as good as anyone else could at a certain level.

  28. #28
    Join Date
    Feb 2011
    Posts
    580
    Quote Originally Posted by BuffaloBill View Post
    HostVirtual actually have their own routers and virtual instances so there is no reason why they could not offer their own IP Anycast services / solution. I think they even have IPv6, so you could get that anycasted as well.

    Just because a company uses DNS Made Easy doesn't mean they don't know how to set up a network that uses IP Anycast. They might just not need to worry about DNS. You have to understand that DNS Made Easy specializes in DNS and is optimized in speed for their clients. This is all they do.
    http://solvedns.com/dns-comparison/2013/10

    HostVirtual offers a different service. But they could probably offer an IP Anycast solution as good as anyone else could at a certain level.
    I personally don't buy this theory. If I were trying to get some specialized service I would rather get it from somebody who does it as their core business rather than somebody who "could probably offer it".

  29. #29
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    Quote Originally Posted by DMDM View Post
    Peer1 is not great for anycast by any stretch of imagination simply because they are a Tier 2 network. Same applies to bigger networks like nlayer. There are couple reasons for that:
    Quote Originally Posted by DMDM View Post
    1. They have assymetric uplinks at their locations. For example look at their Vancouver routing for Level3 customers and compare that to Toronto...
    They have symmetric upstreams in all of their US and EU locations. The Canadian locations are the exceptions given the lack of presence of some of the US providers, so they end up backhauling to the US cities for traffic from those providers, while receiving traffic directly from other Canadian networks there. There is no performance drawbacks to receiving the traffic at the nearest American city and carrying the traffic on their own backbone up to Canada versus having those transit providers extend POPs into all those Canadian cities and handing off traffic there. For the most part, PEER1 has peering with all the other Canadian networks as well (on/off for Bell), so there aren't any cases where traffic is looping out to another city and back.

    For most intents and purposes, Vancouver is just an extension of Seattle. From that perspective, it has the same standard set of transits as any of their other cities. They did pick up a link to Tinet directly in Vancouver, but it wouldn't get used for any domestic Canadian traffic so it's of no consequence. It just allows them to dump traffic out Tinet right away instead of carrying it down to Seattle first over their own backbone. Tinet will still be carrying the traffic down through Seattle anyhow. Inbound traffic from Tinet will still primarily come in through Seattle for Vancouver, and unlikely to change even if Tinet's peering in Canada improves unless PEER1 ends up de-peering with the other Canadian ISP's.

    Do you have any examples of suboptimal routing to Level3 customers specific to how they have Vancouver connected?

    Quote Originally Posted by DMDM View Post
    2. As Tier 2 networks they have transit uplinks. This means that their transit providers setup their routers to prefer the routes of their customers irrespective of any other parameters.
    Firstly, given that they have links to all of their transit providers at pretty much all locations, it's not of any consequence that customer routes are preferred over peering routes. Secondly, as there is a direct peering relationship between provider and customer, I fail to see how preferring a peering route over a customer route would be to any advantage.I might agree with you if they didn't have their own backbone, but that is not the case. Yes, they don't have the best northwest-to-southeast or northeast-to-southwest latency since they don't have a POP in or fiber through Phoenix, but they're by and large on the lowest latency fiber paths between cities otherwise, so you could certainly do a lot worse.

    Quote Originally Posted by DMDM View Post
    The only way to properly balance anycast is to mix Tier 1 providers at each location. Ideally you would use Tier 1 providers that do not ignore the routing policy attributes that your routers export.
    I disagree. A Tier2's backbone acts as a sort of buffer to help balance the anycast. Now, I'd be hesitant to mix Tier2's, not because of inherent performance drawbacks but simply because there'd be a larger number of permutations of ingress paths that would make troubleshooting more difficult. But I'd definitely take anycast through a Tier 2 network like PEER1 over setting up a Tier3 type network with multiple Tier 1 upstreams at every location without any interconnecting backbone like Internap. Also, I won't mention any specifics as it's not public information, but I do know the proportion of PEER1's peering vs transit. I'll just say that it's significant.

    The only issue is when we need locations that they're not in, like Asia. However, given how disparate Asia's networks are from the rest of the Internet, there's the possibility that traffic could just stay within Asia without taking any detours to other parts of the world, with the right mix of transits/peering. We don't yet have enough business interests in Asia to find and buy that mix ourselves, but if someone else already has it worked out (I would have to see clear, specific evidence to back it up before I believe it), we'd be interested in working with them. A perfunctory glance at HostVirtual left me skeptical that they are able to offer that mix (I'm kind of skeptical in general though), due to the lack of any substantive details on their website about their anycast capabilities, but I also try to keep an open mind and if Mark backs up his claims with something more concrete than what's been mentioned so far, they could definitely be an option for us.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  30. #30
    Join Date
    Feb 2011
    Posts
    580
    Quote Originally Posted by hhw View Post
    They have symmetric upstreams in all of their US and EU locations.
    Here are just some examples demonstrating that they do not have symmetric upstreams across US locations (and it should not be expected of them because their business would suffer if they did it). These examples are based on an IP announced by Comcast and these are taken from peer1 looking glass. As you can see Peer1 will route traffic via Level3, Tinet, Tata or directly to Comcast depending on these examples. Notice I did not even include Europe in this sampling despite your claim that upstreams are symmetric across all US and EU locations. If you look from the other side things get even more interesting- this is where customer route preference starts to really defeat anycast.

    Router: Seattle, WA
    Command: traceroute 68.87.26.10

    1 xe-10-0-0.edge1.Seattle3.Level3.net (4.59.232.1) 0.392 ms 0.408 ms 0.366 ms
    2 4.68.105.13 (4.68.105.13) 24.193 ms ae-2-52.edge2.Seattle3.Level3.net (4.69.147.171) 0.448 ms 4.68.105.13 (4.68.105.13) 0.586 ms
    3 comcast-level3.Seattle3.level3.net (4.68.63.66) 5.492 ms 3.347 ms 3.764 ms
    4 he-0-15-0-0-cr01.denver.co.ibone.comcast.net (68.86.86.213) 27.227 ms 29.536 ms 27.747 ms


    Router: San Jose, CA
    Command: traceroute 68.87.26.10

    Tracing the route to www.wcprdp.comcast.com (68.87.26.10)

    1 ix-11-3-0-0.tcore1.SQN-SanJose.as6453.net (66.110.8.49) 8 msec 4 msec 4 msec
    2 if-3-2.tcore2.LVW-LosAngeles.as6453.net (63.243.205.14) [AS 6453] 20 msec 16 msec 12 msec
    3 xe-1-2-0-0-pe01.onewilshire.ca.ibone.comcast.net (173.167.59.185) [AS 7922] 12 msec


    Router: Miami, FL
    Command: traceroute 68.87.26.10

    1 ae3-215.mia10.ip4.tinet.net (77.67.70.213) 15.222 ms 24.074 ms 0.347 ms
    2 te-0-9-0-1-pe01.nota.fl.ibone.comcast.net (66.208.233.17) 2.425 ms 2.857 ms 3.858 ms


    Router: Chicago, IL
    Command: traceroute 68.87.26.10

    1 te-0-6-0-3-pe03.350ecermak.il.ibone.comcast.net (173.167.58.9) 0.725 ms 0.945 ms 0.572 ms

  31. #31
    Join Date
    Oct 2002
    Location
    Vancouver, B.C.
    Posts
    2,656
    None of the examples you've provided actually show a suboptimal path, just that different transits are used for the same IP. It's not really important that the egress path isn't always through the same transit provider. For the purposes of anycast, the ingress path is where symmetry matters, not the egress path, and only that the path is available with all of the transit providers. Unfortunately PEER1's looking glass doesn't give a complete picture because it's somewhat arbitrary which of the two routers in a given city is used, and so not all the transit paths will be visible as only the best path on one router will be redistributed to the other. If you look at the BGP for that IP on Comcast in San Jose for example though, you'll see routes from all 3 of their transits, Level3, TATA, and Tinet. These are the same 3 they have everywhere, which is the symmetry I was referring to. No matter which of the 3 you traceroute from to an anycast IP, you'll go straight to them in that same or nearest city. Which provider is used on the egress path back isn't important so long as it's a good route.

    The only thing they don't have symmetry on here is peering to Comcast, which they turned up after I left so I'm not 100% clear on which cities they're peering in and how they're announcing to them. PEER1 does quite a bit of traffic engineering though, all manual, so there may or may not be any suboptimal paths. Testing would need to be done from Comcast to PEER1's anycast IPs to determine that. I don't think PEER1's CDN is used enough anymore where suboptimal paths would necessarily be reported to them, but hopefully as our CDN gets traction, we'll receive such reports and pass them on to them to get any path issues corrected. That we can do that makes them pretty easy and great to work with. Being a past employee aside, I worked with most of the Tier1 networks over my years there, and have my fair share of horror stories to tell, and I can tell you I much prefer dealing with PEER1 than any of the Tier1's myself.
    Last edited by hhw; 11-19-2013 at 01:15 PM.
    ASTUTE HOSTING: Advanced, customized, and scalable solutions with AS54527 Premium Canadian Optimized Network (Level3, PEER1, Shaw, Tinet)
    MicroServers.io: Enterprise Dedicated Hardware with IPMI at VPS-like Prices using AS63213 Affordable Bandwidth (Cogent, HE, Tinet)
    Dedicated Hosting, Colo, Bandwidth, and Fiber out of Vancouver, Seattle, LA, Toronto, NYC, and Miami

  32. #32
    Join Date
    May 2013
    Location
    Dubai, UAE
    Posts
    280
    Quote Originally Posted by Rebies View Post
    Update... HostVirtual is the first company to come back to me with options. They have a handful of options, if using our own IP space it would be as little as $5k to $15k a year, depending on how many locations we were to use. Anyone else have suggestions on companies that have the backbone and setup to allow us to do this? I have not heard back from a lot of companies, but I expect to early next week.
    HostVirtual does this very well. We're currently using them for our DNS setup.

Similar Threads

  1. What is Anycast DNS ?
    By boosters in forum VPS Hosting
    Replies: 4
    Last Post: 12-11-2012, 11:23 AM
  2. IP Anycast service
    By Jasper852 in forum Dedicated Server
    Replies: 30
    Last Post: 06-05-2011, 01:51 AM
  3. dns anycast?
    By kassoff in forum Colocation and Data Centers
    Replies: 45
    Last Post: 12-09-2010, 07:11 AM
  4. Anycast - Any providers
    By Steven in forum Dedicated Server
    Replies: 7
    Last Post: 05-05-2010, 03:28 AM
  5. IP Anycast (or the like)?
    By Tobarja in forum Hosting Security and Technology
    Replies: 0
    Last Post: 04-16-2005, 03:49 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •