Skip to content
Log in Sign up My account

3.3 High Availability for the Enterprise WAN

Blueprint 3.3: High availability for enterprise WAN

  • 3.3.a Single-homed
  • 3.3.b Multihomed
  • 3.3.c Backup connectivity
  • 3.3.d Failover

A WAN design has to survive link failures, router failures and sometimes whole-provider failures. The art is choosing how much redundancy to buy for each site, and making sure failover is fast enough to matter. Link redundancy at the campus edge uses the same ideas as ENCOR design and HA. BFD mechanics are in ENARSI BFD. BGP policy is in ENARSI BGP policies.

Homing models

Terms vary between vendors and books. The usage below is the common one, so state your assumption in an exam answer if the wording is unclear.

Model Description Survives link failure Survives router failure Survives provider failure
Single-homed One router, one link, one provider No No No
Dual-homed (same provider) Two links to the same provider, ideally to different PE routers Yes Only with two CEs No
Multihomed Links to two or more providers Yes Only with two CEs Yes
Dual-router multihomed Two CEs, each with its own provider and link Yes Yes Yes 
graph TB
  subgraph SH ["Single-homed"]
    A1["CE"] --- P1["Provider A"]
  end
  subgraph DH ["Dual-homed"]
    A2["CE"] --- PA1["Provider A PE1"]
    A2 --- PA2["Provider A PE2"]
  end
  subgraph MH ["Multihomed, dual router"]
    B1["CE1"] --- PX["Provider A"]
    B2["CE2"] --- PY["Provider B"]
    B1 --- B2
  end

Matching the model to the site

Site type Common choice
Small branch, tolerant of outage Single-homed, with cellular backup
Medium branch Single CE, two transports (MPLS plus Internet), cellular optional
Large branch or regional hub Two CEs, two providers, path-diverse circuits
Data centre or headquarters Two CEs per provider, at least two providers, diverse entrances

Failure domains and shared fate

Two circuits from two providers are not redundant if they:

  • Enter the building through the same duct.
  • Share the same last-mile provider (many carriers lease from the same local exchange carrier).
  • Terminate on the same PE router or in the same provider facility.
  • Land on the same CE, line card or power supply.

Always ask for physical path diversity and separate demarcation. This is the sort of thing a design question tests with the phrase "both providers use the same conduit".

Backup connectivity

Backup type Notes
Secondary wired circuit (same or another provider) Best performance. Costs a full circuit.
Internet VPN backup for an MPLS site Cheap and common. Needs an encrypted overlay. Bandwidth and latency differ from the primary.
4G/5G cellular Fast to deploy, metered. Good for small sites. See 3.1.
Active/active over two transports Both carry traffic. Better use of money, but you must plan capacity so one link can carry the load after a failure.

Capacity after failure

A common mistake is sizing both links to carry 70% each. When one fails, the survivor has to carry 140% of its capacity. Active/active designs need each link sized to hold the traffic that matters (usually voice and critical data) alone, with QoS dropping the rest.

Failover mechanisms

How fast the WAN fails over depends on how the failure is detected.

Detection method Typical speed Notes
Interface down (physical loss of carrier) Milliseconds to a second Does not detect failures beyond the first hop (provider-side faults)
Routing protocol hello/dead timers OSPF 40 s, EIGRP 15 s or 60 s, BGP 180 s by default Too slow for voice unless tuned
BFD Sub-second Works for OSPF, EIGRP, BGP, static routes. Best general tool
IP SLA with object tracking Seconds Works with static routes and PBR. Tests the end-to-end path
Tunnel keepalives and DPD Seconds to tens of seconds Needed on GRE and IPsec overlays because the underlay can fail silently
SD-WAN BFD on tunnels Seconds, configurable Also measures loss, latency and jitter for app-aware routing

Notes on the defaults: OSPF hello is 10 s and dead is 40 s on broadcast and point-to-point networks. EIGRP hello is 5 s and hold is 15 s on most links, but 60 s and 180 s on low-speed multipoint links. BGP keepalive is 60 s and hold is 180 s.

Floating static route with IP SLA

This is the standard backup design when the primary is a link without a routing protocol (Internet DIA, cellular).

ip sla 10
 icmp-echo 198.51.100.1 source-interface GigabitEthernet0/0/0
 frequency 5
ip sla schedule 10 life forever start-time now
track 10 ip sla 10 reachability
!
ip route 0.0.0.0 0.0.0.0 198.51.100.1 track 10
ip route 0.0.0.0 0.0.0.0 203.0.113.1 250

The primary default is removed when the probe fails. The second default has administrative distance 250, so it is only used when the first is absent. Probe a target beyond the first hop, such as a provider router or a well-known address, otherwise the check only proves that the local link is up.

BGP for dual-ISP internet edge

When the enterprise has its own AS and prefix, multihoming to two ISPs is done with BGP.

Question Typical answer
Which routes to accept from ISPs? Default route only (simple), default plus partial table (better outbound steering), or full table (most control, needs a large router)
How to steer inbound traffic? Advertise more specific prefixes on one link, or AS-path prepend on the backup link. Local-preference does not influence the other AS
How to steer outbound traffic? Local preference on routes from the preferred ISP
How to avoid becoming a transit AS? Filter outbound so only your own prefixes are advertised 
ip prefix-list MY-PREFIXES seq 5 permit 192.0.2.0/24
!
route-map TO-ISP permit 10
 match ip address prefix-list MY-PREFIXES
!
router bgp 65010
 neighbor 198.51.100.1 remote-as 64500
 neighbor 198.51.100.1 route-map TO-ISP out
 neighbor 203.0.113.1 remote-as 64501
 neighbor 203.0.113.1 route-map TO-ISP out

If the outbound filter is missing, the enterprise learns ISP A's routes and advertises them to ISP B (and the reverse), and becomes a transit path for Internet traffic. This is a classic design error. See ENARSI BGP policies for filtering and attribute detail, and ENARSI BGP best path for path selection.

Router redundancy on the LAN side

When the site has two WAN edge routers, the LAN needs a gateway that survives the loss of either one. Use an FHRP (HSRP or VRRP) with tracking of the WAN interface or the IP SLA state, so the LAN gateway moves when the WAN side fails. Without tracking, the active gateway can stay up with a dead uplink and black-hole traffic. See ENCOR FHRP.

Between the two edge routers run a routing protocol or an iBGP session so that if one WAN link fails, the other router can carry the traffic.

Asymmetric routing and stateful devices

When both links are active, traffic can leave through one router and return through the other. A stateful firewall or NAT device in only one path will drop the return flow. Options:

  • Keep flows symmetric using routing policy (local preference or PBR) and per-flow hashing consistent on both ends.
  • Use a stateful HA pair at the edge so both paths terminate on one logical device.
  • Use SD-WAN, where the fabric controls path selection and can keep flows on one path.

Convergence on overlays

An overlay hides the underlay failure. A GRE or IPsec tunnel stays up until its keepalive fails, so tune keepalives or use DPD, and run BFD on the tunnel. DMVPN with two hubs relies on routing-protocol metrics to prefer the primary hub and on NHRP registration for the secondary hub to learn the spoke.

Design exercise

Scenario. A bank has 60 branches. Each branch has an MPLS L3VPN circuit with provider A and a broadband Internet circuit. The branches run voice and core-banking traffic. Internet browsing must never use the MPLS circuit. The head office connects to both providers and also to Internet for its own public services using its own AS and provider-independent prefix. Requirements: voice calls must survive a link failure with less than 5 seconds of interruption, the head office must not become a transit AS, and a branch outage of an hour affects customers.

Requirements and constraints.

  • Branch failover under 5 seconds.
  • Head office Internet edge uses two ISPs with its own AS.
  • No transit.
  • Branch Internet backup must not use the more expensive MPLS capacity for browsing.
Model answer
  • Branch: one CE with two transports. A DMVPN or IPsec VTI over the broadband circuit backs up the MPLS path. Run BFD on the PE-CE session for the MPLS circuit and on the tunnel (or tune the tunnel keepalive) so failure detection is in the one-to-three second range, well under 5 seconds. Prefer MPLS for voice and core banking, and use the Internet circuit for guest and browsing traffic with local breakout.
  • Larger branches: add a second CE and an FHRP with tracking toward the WAN, because a single CE is a single point of failure that the question's one-hour outage tolerance rules out.
  • Head office Internet: BGP multihoming with both ISPs, accept default plus partial routes, steer outbound with local preference, steer inbound by prepending on the backup ISP. Apply an outbound prefix-list filter so only the bank's own prefix is advertised.
  • Rejected: relying on OSPF or BGP default timers (40 s and 180 s) misses the 5-second target. A floating static route with IP SLA only is acceptable for the Internet branch path but is slower than BFD for the MPLS path. Two providers sharing the same last-mile conduit would not give true redundancy, so the contract should require diverse paths.

Exam trap

Detecting a carrier-side failure needs more than an interface-down event. A provider failure that leaves the local interface up will not trigger anything unless BFD, IP SLA or a protocol timer notices. When the scenario says "the link stays up but traffic is lost", the answer is BFD or IP SLA tracking, not a faster interface.

Exam trap

Multihoming to two ISPs without an outbound filter makes the enterprise a transit AS. The correct fix is to advertise only your own prefixes (prefix-list or AS-path filter) on the outbound policy, not to prepend more.

Exam trap

AS-path prepending and MED influence inbound traffic from other ASes. Local preference influences outbound traffic from your AS. Local preference is not sent to external neighbours.

Quick check

A branch has an MPLS and an Internet circuit on one CE. The MPLS provider has an outage but the CE interface stays up. What makes the router fail over fast?

BFD on the PE-CE routing session, or an IP SLA probe beyond the first hop tracked by a static route. The interface state alone does not change because the fault is inside the provider.

Why track the WAN link in HSRP on a dual-edge site?

Without tracking, the HSRP active router stays active after its WAN link fails, and LAN hosts send traffic to a gateway with no path out. Tracking lowers the priority so the standby takes over.

Why is a second circuit from a different provider sometimes not truly redundant?

Both can share the same duct, last-mile carrier or building entrance, so one dig-up can cut both. Ask for physical path and demarcation diversity.

References

  • Cisco Live On-Demand Library: search "WAN high availability", "Internet edge design", "BFD"
  • Cisco Enterprise Internet Edge and WAN design: search "Enterprise Internet Edge Design Guide" on cisco.com
  • RFC 5880 (BFD), RFC 4271 (BGP-4), RFC 7454 (BGP operations and security)