Skip to content
Log in Sign up My account

Hybrid Cloud Connectivity Design

This article covers how to connect an on-premises network to AWS, Azure and Google Cloud: the private circuit options (Direct Connect, ExpressRoute, Cloud Interconnect), IPsec VPN as the baseline or backup, and SD-WAN cloud on-ramps. You will get the common design patterns, the redundancy rules that actually matter, and the BGP decisions you will make on every one of these links.

The four building blocks

Option Transport Typical use Trade-off
AWS Direct Connect Private circuit to an AWS location, BGP Predictable latency, high volume Lead time, cost of circuits and cross-connects
Azure ExpressRoute Private circuit via a provider or ExpressRoute Direct, BGP Same, for Azure and Microsoft 365 services Same
Google Cloud Interconnect Dedicated, Partner or Cross-Cloud Interconnect, BGP via Cloud Router Same, for GCP Same
IPsec VPN Encrypted tunnels over the internet Quick start, backup, small sites Variable latency, per-tunnel throughput limits
SD-WAN on-ramp Overlay from branches to a cloud-hosted virtual edge Many branches, one policy model Another control plane to operate

The private circuits are not encrypted by default. If your policy requires encryption in transit, run IPsec over the circuit, use MACsec where the service supports it, or encrypt at the application layer. Check each provider's current documentation for what is offered on which port types.

AWS Direct Connect

Direct Connect gives you a dedicated connection (a physical port at a Direct Connect location) or a hosted connection (a slice provisioned by a partner). On top of the connection you create virtual interfaces (VIFs), each one a VLAN with its own BGP session:

  • Private VIF: reaches VPCs through a virtual private gateway or a Direct Connect gateway.
  • Transit VIF: reaches one or more Transit Gateways through a Direct Connect gateway.
  • Public VIF: reaches AWS public service endpoints using AWS public address space.
graph LR
    DC["On-prem DC<br/>AS 65001"] --- R1["Router A"]
    DC --- R2["Router B"]
    R1 -- "DX connection 1<br/>Location X" --> DXGW["Direct Connect gateway"]
    R2 -- "DX connection 2<br/>Location Y" --> DXGW
    DXGW --> TGW["Transit Gateway"]
    TGW --> VPC1["VPC prod"]
    TGW --> VPC2["VPC shared services"]

Design notes that cause real problems if missed:

  • The Direct Connect gateway is a global object. It connects the VIFs to gateways in other regions, but it does not route between your VPCs, and it does not pass traffic from one on-premises site to another. Do not treat it as a transit hub for site-to-site traffic.
  • Prefix limits apply in both directions on the BGP sessions. Summarize before you advertise and check the current quotas for your VIF type.
  • Use BGP authentication on every session and BFD for fast failure detection where supported.

Azure ExpressRoute

An ExpressRoute circuit always comes as a pair of links to two Microsoft edge routers. The provider delivers two BGP sessions per peering and you are expected to run both. Peerings:

  • Private peering reaches VNets through an ExpressRoute virtual network gateway.
  • Microsoft peering reaches Microsoft 365 and Azure public services (PaaS) with public IP space and NAT requirements. Check current guidance before planning for Microsoft 365 over ExpressRoute, as it requires approval and specific design.

Microsoft's side uses AS 12076 for ExpressRoute. Each peering needs a pair of /30 subnets (primary and secondary link) for IPv4. ExpressRoute Global Reach connects two of your circuits through the Microsoft backbone, which lets two on-premises sites in different metros exchange traffic over Microsoft's network. Treat that as a deliberate design choice, not a default, since it affects your traffic path and cost.

Gateway SKU selection limits throughput and features such as FastPath, so size the gateway to the circuit and check the current SKU table.

Google Cloud Interconnect

Interconnect has three forms:

  • Dedicated Interconnect: your own physical connection at a colocation facility.
  • Partner Interconnect: a service provider delivers the connection and you attach through it.
  • Cross-Cloud Interconnect: a direct link between Google and another cloud provider.

Each connection carries one or more VLAN attachments. Each attachment is tied to a Cloud Router, which runs BGP with your on-premises router. Google publishes specific topologies (for example, two connections in each of two metros) for higher availability SLAs. Follow the published topology exactly if you want the SLA, because a design that merely looks redundant may not qualify.

Cloud VPN and IPsec

All three providers offer IPsec gateways with BGP support:

  • AWS Site-to-Site VPN gives two tunnels per connection, terminating on two endpoints in different availability zones. Use both. Tunnel throughput is capped, so check the current limit and use ECMP on a Transit Gateway if you need more aggregate throughput.
  • Azure VPN Gateway should be deployed active-active with BGP enabled for a resilient design.
  • Google HA VPN has two interfaces and requires a tunnel on both to qualify for its availability SLA.

VPN is useful in three roles: initial connectivity while a circuit is being built, a backup path for a private circuit, and the primary path for small or low-criticality sites. As a backup, it only works if you have tested the failover, and if the BGP policy prefers the circuit.

SD-WAN cloud on-ramp

An SD-WAN on-ramp places a virtual edge router (or a pair) in a cloud VPC or VNet, and the SD-WAN overlay extends to it. Branch sites reach cloud workloads through the overlay and the same application policy as everywhere else. Cisco, and other SD-WAN vendors, package this with automation that creates the cloud-side transit constructs for you.

Where it fits: many branches, existing SD-WAN fabric, a need for consistent segmentation and policy across WAN and cloud. Where it does not: one data centre with heavy bulk transfer, where a dedicated circuit to the cloud is simpler and cheaper per gigabit. Many networks use both: circuits for the data centre, SD-WAN on-ramp for branches.

Design patterns

Pattern 1: dual circuit, dual location

Two circuits at two different provider locations, terminating on two different on-premises routers. This survives a router, a circuit or a facility failure. A second circuit into the same facility survives much less than it appears to.

Pattern 2: circuit primary, VPN backup

graph TB
    ONP["On-prem edge"] -- "Circuit, preferred" --> CLOUD["Cloud gateway"]
    ONP -. "IPsec VPN, backup" .-> CLOUD

Both paths run BGP. The circuit wins because of higher local preference on-premises and, towards the cloud, a mechanism the provider supports (communities on AWS, AS-path prepending or more-specific prefixes elsewhere). Remember that a VPN has a lower MTU than the circuit, so path MTU and TCP MSS clamping matter on failover.

Pattern 3: hub and spoke in the cloud

One transit hub (Transit Gateway, Azure Virtual WAN or a hub VNet with an NVA, Google Network Connectivity Center) terminates all external connections. Spoke VPCs and VNets attach to the hub. Inspection firewalls sit in the hub. This keeps the number of circuit attachments small and the routing policy in one place.

Pattern 4: multi-cloud through a colocation exchange

Terminate circuits to AWS, Azure and Google in one colocation facility (or via a cloud exchange provider) and run your own routers there. On-premises reaches all clouds over one WAN link into the facility, and cloud-to-cloud traffic stays off the internet. Google's Cross-Cloud Interconnect is the provider-native alternative for the cloud-to-cloud part.

BGP considerations

Every one of these services is a BGP design in the end.

Choose ASNs deliberately. Use a private ASN for the on-premises side or your own public ASN. Cloud sides have default or configurable ASNs; a duplicate ASN in the path causes loop prevention to drop routes. Check each provider's ASN rules when you create the gateway, as some cannot be changed later.

Control inbound and outbound paths separately. Outbound from on-premises is yours to control with local preference. Inbound towards the cloud depends on what the provider honours:

  • AWS Direct Connect honours local preference communities on private and transit VIFs: 7224:7100 (low), 7224:7200 (medium), 7224:7300 (high). Confirm the current list before relying on it.
  • Elsewhere, expect to use AS-path prepending, MED where honoured, or advertising more-specific prefixes from the preferred path.

Prefer longest match as a tool. If the circuit advertises a /24 and the VPN advertises the covering /23, the circuit wins in the cloud regardless of attributes. That is a predictable failover, but only if the /24 is withdrawn when the circuit fails. Test that your router withdraws it.

Watch for asymmetry. Stateful firewalls on the cloud or on-premises side drop return traffic that arrives on a different path than the outbound traffic. Either keep policy symmetric (same preference both ways) or use active/active designs that share state.

Summarize and filter. Send the cloud a small set of summaries, accept only expected prefixes back, and set maximum-prefix limits. An accidental default route or a leaked internet table through a cloud gateway is a very bad day.

Overlapping address space. Cloud and on-premises ranges must not overlap on a routed design. If they do, you need NAT, and every NAT in the path complicates troubleshooting. Plan IP addressing before the first VPC exists.

router bgp 65001
 neighbor 169.254.10.1 remote-as 64512
 neighbor 169.254.10.1 password <secret>
 neighbor 169.254.10.1 fall-over bfd
 !
 address-family ipv4
  neighbor 169.254.10.1 activate
  neighbor 169.254.10.1 prefix-list FROM-CLOUD in
  neighbor 169.254.10.1 prefix-list TO-CLOUD out
  neighbor 169.254.10.1 maximum-prefix 100 80
 exit-address-family

The addresses and ASN above are illustrative. Use the values the provider generates for your connection.

Redundancy checklist

  • Two circuits, at two locations, on two on-premises routers.
  • Ask your provider for diverse physical paths, in writing, and confirm that the two circuits are not on one fibre or one conduit.
  • A tested VPN backup, with the BGP policy that prefers the circuit.
  • BFD on the BGP sessions where supported, with timers the provider accepts.
  • Monitoring for BGP state, prefix counts and circuit light levels or utilisation. Alert on prefix count changes as well as session down.
  • A scheduled failover test, at least once a year, and after any routing policy change.

Warning

A pair of circuits that share a provider, a building entrance or a router has a single point of failure that no diagram shows. Ask where the fibres run.

Choosing between the options

Start from requirements: bandwidth, latency tolerance, security policy, lead time and number of sites. If the answer is "a few Mbps, tomorrow", it is VPN. If it is "steady multi-gigabit data replication", it is a private circuit with a VPN backup. If it is "200 branches that need cloud apps with one policy", it is SD-WAN with an on-ramp. Most large networks end up with all three.

For the BGP fundamentals that sit under all of this, the CCNP course covers eBGP, route filtering and path attributes at cisconets.com/ccnp, and the lab setup page shows how to build a practice topology.

References