Skip to content
Log in Sign up My account

3.5 Cisco SD-WAN Design Considerations

Blueprint 3.5: Cisco SD-WAN design considerations

  • Control plane design
  • Overlay design
  • LAN design
  • High availability and redundancy
  • Scalability
  • Security design
  • QoS and multicast over the SD-WAN fabric

This page assumes you know the components from 3.4 and the platform basics from the ENCOR SD-WAN page. The question here is how to arrange them.

Control plane design

Decision Options Guidance
How many Controllers Two minimum, more for scale Place them in different locations or availability zones. Each edge connects to a subset of them.
Controller affinity Groups so that edges prefer certain Controllers Use it to spread load and to keep an edge on a nearby Controller. Define a primary and a backup group.
How many Validators Two or more, reachable by name Reached through DNS, so multiple IPs behind one name give redundancy.
Manager Cluster of nodes Cluster for scale and for survivability of operations.
Deployment Cisco cloud-hosted, partner or on-premises Cloud-hosted for lower operating effort. On-premises for sovereignty or isolation.

Place Controllers where every edge can reach them over at least two transports. If an edge has MPLS and Internet, the Controllers need to be reachable over both, or the edge loses control connections when one fails. Controllers on public IP addresses are reached over the Internet. Reaching them over MPLS needs a private address or a gateway. This is a common design gap.

Overlay design

Three questions define the overlay.

1. Topology. Fully meshed, hub-and-spoke, regional, or a mix.

Topology Strength Weakness
Full mesh Direct paths, lowest latency Tunnel count grows with the square of the sites and colours. Edge platform limits are reached.
Hub-and-spoke Simple, central inspection and services Hub is a bottleneck and a latency penalty for spoke-to-spoke traffic
Regional hubs (hierarchical) Contains scale, supports regional inspection More policy and more design work
On-demand tunnels Spokes build direct tunnels only when traffic needs them Slight first-packet delay, needs supported release and platforms

Tunnel count is the real scaling limit on an edge, not the routing table. Reduce it by:

  • Restricting colours so MPLS only talks to MPLS and Internet only to Internet. Without it, every pair of colours can form a tunnel.
  • Control policy to turn a full mesh into hub-and-spoke by controlling TLOC advertisement.
  • Multi-Region Fabric, which splits the network into regions with border routers between them and reduces the number of tunnels per edge.
  • On-demand tunnels for large branch counts.
graph TB
  subgraph R1 ["Region 1"]
    A1["Branch"] --- BR1["Border router"]
    A2["Branch"] --- BR1
  end
  subgraph R2 ["Region 2"]
    B1["Branch"] --- BR2["Border router"]
    B2["Branch"] --- BR2
  end
  BR1 == "Core overlay" === BR2
  DC["Data centre hub"] --- BR1
  DC --- BR2

2. Transport mix. Typically MPLS plus broadband Internet, sometimes with LTE. Colour naming and restrict choices must reflect it.

3. Service insertion. Where do firewalls, IPS and Internet breakout live? Options are branch local breakout, regional hubs, a data centre, or a cloud security service. A service chain can insert a firewall in the path for selected VPNs.

LAN design

The service side (LAN) of the edge connects to the campus or branch network.

Choice Notes
Routing to LAN Static, OSPF, EIGRP or BGP. Choose what the site already runs and keep it simple.
Redistribution Edges redistribute between OMP and the LAN protocol. At a site with two edges this can create loops.
Dual-edge sites Both edges have the same site ID. Use OSPF route tags or BGP communities and filters to stop routes learned from OMP being sent back into OMP by the other edge.
LAN gateway redundancy Use VRRP on the service side. Track the WAN state so the gateway moves when the WAN side fails.
Sharing a transport A TLOC extension lets one edge use the other edge's WAN circuit through a link between them. 
graph TB
  subgraph Site ["Dual-edge site, one site ID"]
    E1["Edge 1"] --- E2["Edge 2"]
    E1 -- "MPLS" --- T1((MPLS))
    E2 -- "Internet" --- T2((Internet))
    E1 -. "TLOC extension" .- E2
    E1 --- L["LAN switch pair<br/>VRRP gateway"]
    E2 --- L
  end

Plan LAN-side addressing and summarisation. OMP can carry summaries, which reduces table size at small edges. Keep VPN numbering consistent across sites, because the same VPN number means the same segment everywhere.

High availability and redundancy

Layer Mechanism
Transport Two or more transports per site. Ideally path-diverse.
Device Two edges per site at critical sites, with VRRP or routing on the LAN.
Control At least two Controllers and two Validators in different locations.
Management Manager cluster. Backups of the Manager database.
Graceful restart OMP graceful restart keeps the last known state if controllers are unreachable for a time.
Failure detection BFD in each tunnel. Tune the hello interval and multiplier to the needed detection time, balancing speed with false positives on a noisy Internet link.

An important point: application-aware routing (AAR) moves traffic away from a path that violates its SLA even if the path is "up". That is different from simple failover, and it is one of the reasons to prefer SD-WAN for mixed transports.

Scalability

Look at four limits:

  1. Controller control connections. Each edge holds control connections to the Controllers. Use more Controllers and affinity to spread them.
  2. Edge tunnel count. Reduced by colour restriction, hub-and-spoke control policy, regional design and on-demand tunnels.
  3. Edge route table and throughput. Pick the platform by encrypted throughput with the features turned on (firewall, IPS, QoS), not by interface speed.
  4. Manager scale. Statistics, number of devices and API load drive the cluster size.

Always check the current scaling guide for supported numbers per release and platform instead of relying on a fixed figure.

Security design

  • Segmentation. Map business units or risk zones to VPNs. Keep guest and IoT in separate VPNs. Leak between them only through a firewall or controlled policy.
  • Direct Internet access (DIA). Breakout at the branch cuts latency and WAN cost for SaaS but puts the branch on the Internet, so add edge firewall, IPS and DNS security or send the traffic to a cloud security service.
  • Controller exposure. Controllers and Validators need to be reachable. Restrict by ACL to the needed ports and keep them patched.
  • Certificates. Decide who is the CA (Cisco, an enterprise CA) and plan certificate lifetime and renewal.
  • Data-plane encryption. On by default for tunnels between edges. Rekey intervals and cipher suites are configurable.
  • Cloud on-ramp. Edges in the cloud (or a cloud gateway) extend segments to IaaS and give SaaS path selection. See blueprint 5.5 in the Automation module for cloud connectivity patterns.

QoS over the SD-WAN fabric

SD-WAN does not remove the need for QoS. It adds structure.

  • Mark and classify traffic at the LAN edge (trust DSCP from trusted devices, re-mark otherwise). See the QoS toolkit in ENCOR QoS for the MQC building blocks.
  • Shape egress on each WAN interface to the real circuit rate. With a 1 Gbps port in front of a 100 Mbps circuit, queuing only works if you shape first.
  • Use low-latency queueing for voice and class-based queues for the rest. Cisco SD-WAN edges use a fixed set of eight queues (queue 0 is the low-latency queue) mapped from forwarding classes.
  • Map to the provider's classes on MPLS and keep DSCP intact inside the IPsec packet (the DSCP is copied to the outer header).
  • Application-aware routing uses SLA classes (loss, latency, jitter) per application to pick a path. QoS governs treatment on a path. AAR chooses the path.
  • The IPsec overhead reduces the usable MTU. Set TCP MSS adjustment to avoid fragmentation.

Multicast over the fabric

The overlay is unicast IPsec, so multicast needs support in the edge.

  • Edges run PIM towards the LAN and carry multicast state in OMP (multicast routes).
  • An edge or hub acts as a replicator: it receives one multicast stream and copies it as unicast to each interested remote edge. Pick replicators with enough bandwidth, ideally at hubs or data centres.
  • The multicast source and the RP should be placed with the replicator in mind. A source far from a replicator adds a hairpin.
  • Limit the multicast groups in use and the number of receivers per replicator.

For multicast fundamentals see ENCOR multicast.

Exam trap

Dual-edge sites need loop prevention when routes pass between OMP and the LAN IGP. If an exam stem says routes from the overlay are being re-advertised back into the overlay by the second edge, the fix is route tagging or filtering on redistribution, not a different overlay topology.

Exam trap

Full mesh is not the default answer for "best performance". Tunnel count is the limit on an edge. For large networks, regional hubs, restricted colours, Multi-Region Fabric or on-demand tunnels are the scalable answers.

Exam trap

Shape before you queue. On a high-speed handoff in front of a slower circuit, LLQ and class queues do nothing until a shaper makes congestion appear on the router.

Design exercise

Scenario. A global company has 1,200 branches in four regions and three data centres. Each branch has broadband Internet, and 900 also have MPLS. Branches use SaaS heavily and voice runs between branches in the same region. Security requires segmentation of corporate, guest and PoS. Large branches have two edge routers. The edges at the smallest branches are low-end platforms.

Requirements and constraints.

  • Direct SaaS access with security.
  • Regional voice with low latency.
  • Edge tunnel limits on small platforms.
  • Resilience of the control plane across regions.
Model answer
  • Overlay: use a regional design (Multi-Region Fabric or regional hubs). Branches build tunnels inside their region, and borders connect the regions through the core, which keeps tunnel counts within what small platforms support. Allow on-demand spoke-to-spoke tunnels for regional voice if the release supports them. Restrict colours so MPLS to MPLS and Internet to Internet, which halves the tunnel pairs.
  • Control plane: at least two Controllers and two Validators per region using affinity groups, so branches use their regional Controllers first and fail over to another region. Manager cluster in a data centre or the cloud.
  • Security: three VPNs (corporate, guest, PoS) kept apart by default. DIA for SaaS with integrated firewall, DNS security and IPS where the platform allows, or a cloud security service for the smallest edges. The PoS VPN has no Internet breakout.
  • LAN and HA: large branches use two edges with the same site ID, VRRP on the LAN with WAN tracking, and tagging to prevent routing loops between OMP and the LAN IGP.
  • Rejected: a global full mesh exceeds the tunnel limits of small edges. A single hub per region forces SaaS traffic to hairpin and adds latency. Putting all Controllers in one data centre creates a regional single point of failure.

Quick check

Two edges at one site learn each other's routes from OMP and re-advertise them to OSPF. Why is that a problem?

The second edge can take an OSPF route that originated from OMP and advertise it back into OMP as a local route, which creates a loop or suboptimal path. Tag routes on redistribution into OSPF and filter tagged routes when redistributing back into OMP.

How does restricting colours reduce load on an edge?

By default an edge tries to build a tunnel to every remote TLOC. With restrict, a tunnel interface only forms tunnels with TLOCs of the same colour, so an MPLS interface does not try to build tunnels to Internet TLOCs. This cuts the number of tunnels and BFD sessions.

Why does multicast over SD-WAN need a replicator?

The fabric is made of unicast IPsec tunnels, so a single multicast stream must be copied once for each remote edge that has receivers. A replicator does that copying, so it should have enough bandwidth.

References

  • Cisco Catalyst SD-WAN design and scaling: search "Cisco Catalyst SD-WAN Design Guide", "Cisco SD-WAN Multi-Region Fabric" and "Cisco Catalyst SD-WAN Scalability" on cisco.com
  • Cisco Live On-Demand Library: search "SD-WAN design", "SD-WAN scalability", "SD-WAN multicast", "SD-WAN QoS"
  • Cisco DevNet: SD-WAN Manager API documentation