Skip to content
Log in Sign up My account

Spine-leaf design rules: oversubscription, ECMP and failure domains

This article covers the rules that decide whether a spine-leaf fabric behaves well under load and during failures. You will get the oversubscription arithmetic with worked examples, an explanation of how ECMP hashing spreads (and fails to spread) traffic, a comparison of eBGP, OSPF and IS-IS as the underlay, guidance on border and services leaves, the options for scaling past one pod, and a checklist of failure-domain and maintenance practices and common mistakes. It applies to any vendor. The examples use generic port counts so you can substitute your own hardware.

Why spine-leaf

The classic three-tier design (access, distribution, core) was built for north-south traffic: clients talking to servers in a central data centre. Spanning tree blocked redundant links, and traffic between two racks often crossed several tiers with different oversubscription at each one.

Modern data centres carry mostly east-west traffic: storage replication, microservices, VM migration, distributed databases. A spine-leaf (folded Clos) fabric fits this because:

  • Every leaf connects to every spine. Any leaf-to-leaf path is exactly two hops (leaf, spine, leaf), so latency and hop count are predictable.
  • All links are routed and forwarding. Nothing is blocked, so bandwidth scales by adding links.
  • Capacity grows in two independent directions. Add leaves for more server ports. Add spines for more bisection bandwidth.
  • The spine is simple. It carries underlay routes and, in an EVPN-VXLAN fabric, usually only the BGP control plane for overlay reachability. It does not terminate hosts.
graph TB
    S1["Spine 1"] --- L1["Leaf 1"]
    S1 --- L2["Leaf 2"]
    S1 --- L3["Leaf 3"]
    S1 --- L4["Leaf 4"]
    S2["Spine 2"] --- L1
    S2 --- L2
    S2 --- L3
    S2 --- L4
    L1 --- H1["Servers"]
    L2 --- H2["Servers"]
    L3 --- H3["Servers"]
    L4 --- H4["Border / services"]

Two rules define the topology. Leaves never connect to leaves (except a vPC or MLAG pair peer link, which is a local detail). Spines never connect to spines. Breaking either rule creates paths of unequal length and undermines ECMP.

Oversubscription math

Oversubscription at a leaf is the ratio of server-facing bandwidth to uplink bandwidth:

oversubscription = (downlink ports x downlink speed) / (uplink ports x uplink speed)

A ratio of 1:1 is non-blocking. A ratio of 3:1 means that if every server transmits at line rate to other racks at once, each gets one third of its rate.

Worked examples

Example 1: 25G servers, 100G uplinks. A leaf with 48 x 25G server ports and 6 x 100G uplinks.

downlink = 48 x 25G  = 1200G
uplink   =  6 x 100G =  600G
ratio    = 1200 / 600 = 2:1

With 6 uplinks, the fabric needs 6 spines (one uplink per spine). If each spine has 64 x 100G ports, the fabric supports up to 64 leaves, so 64 x 48 = 3,072 server ports.

Example 2: 10G servers, 40G uplinks. A leaf with 48 x 10G and 4 x 40G.

downlink = 480G
uplink   = 160G
ratio    = 3:1

This is a common ratio for general-purpose compute, where average utilisation is well under 30 percent.

Example 3: non-blocking for storage or AI/ML. A 32 x 100G leaf split evenly: 16 ports down, 16 up.

downlink = 16 x 100G = 1600G
uplink   = 16 x 100G = 1600G
ratio    = 1:1

This needs 16 spines. Each spine port count then limits the leaf count. A 32-port spine supports 32 leaves, so 32 x 16 = 512 server ports. Non-blocking costs ports: half of every leaf goes to uplinks.

Choosing a ratio

  • Measure first. Peak-hour 95th percentile utilisation per leaf uplink tells you more than the datasheet.
  • Match the workload. General virtualisation tolerates 3:1 to 4:1. Storage, big data shuffles and GPU clusters often need 1:1 or close to it.
  • Plan for failure, not just the healthy state. With 4 spines and one down, the effective ratio on every leaf worsens by a third (4 uplinks become 3). A 2:1 design is 2.67:1 in that state. A 2-spine fabric degrades to double the ratio when one spine fails.
  • Count the border and services leaves separately. They concentrate traffic from the whole fabric and often need 1:1 or better.
  • Keep uplink speed higher than server speed where you can. Four 100G uplinks serving 25G hosts hash far more smoothly than eight 10G uplinks serving 10G hosts, because a single flow can never exceed one link.

Tip

Design uplink speed to be at least 4x the server port speed. A single elephant flow then fills at most a quarter of one uplink, which keeps hash imbalance tolerable.

ECMP and hashing

With every leaf connected to every spine, a leaf has N equal-cost next hops toward any remote prefix. The forwarding ASIC picks one per packet using a hash of header fields. The hash is per flow, not per packet, so packets of one flow stay in order.

The usual input is the 5-tuple: source IP, destination IP, protocol, source port, destination port. Some platforms add ingress interface, VLAN or a per-device seed.

What this means in practice

  • ECMP balances flows, not bytes. A few large flows can land on the same uplink while others sit idle. Many small flows balance well. This is statistical, so expect 10 to 20 percent variation on a busy link, and worse with few flows.
  • VXLAN preserves entropy only if the encapsulation does. The VTEP builds the outer UDP source port from a hash of the inner headers (see RFC 7348). Spines hash on the outer header, so different inner flows between the same two VTEPs still spread. If a platform fixes the outer source port, all traffic between a VTEP pair lands on one path.
  • Hash polarisation. If every tier uses the same hash function and seed, traffic that was split at the leaf is split the same way at the next tier, so some paths get nothing. In a two-tier fabric this is rare, but it appears with super-spines or when firewalls and routers sit in the path. Most platforms let you change the hash seed or offset per device; use a different value on each tier.
  • Hash inputs must include L4 ports. If a platform defaults to IP-only hashing, a single pair of hosts running many connections uses one path. Enable source and destination port in the load-sharing configuration.
  • Link failure causes re-hashing. Removing a next hop changes the hash modulus, and on simple implementations many flows move. Some platforms offer resilient (consistent) hashing that moves only the flows on the failed link. Check whether yours does, especially in front of stateful services.
  • Maximum paths must be raised. Many platforms default to one or a small number of ECMP paths for BGP. Set it to at least the spine count.
router bgp 65101
  address-family ipv4 unicast
    maximum-paths 8

Verify that the RIB holds all the paths, not only that BGP does. Check the forwarding table too, since a RIB entry with N next hops is useless if the FIB programmed fewer.

ECMP assumes equal-cost, equal-capacity paths. If one uplink is 100G and another is 40G, plain ECMP sends equal flows to both and the slower link congests first. Keep all leaf-to-spine links the same speed. If you must mix, either do not rely on ECMP across them or use a platform feature for weighted ECMP (for example, BGP link bandwidth), and test it.

Underlay choices

The underlay's job is simple: advertise loopbacks (VTEP and router-ID addresses) and point-to-point links so every VTEP can reach every other over ECMP. Three protocols are in common use.

eBGP OSPF IS-IS
Scaling Very large; designed for it Area design needed past a few hundred nodes Large; flat L2 handles big fabrics
Convergence Good with BFD and tuned timers Good with BFD, SPF throttling Good with BFD, SPF throttling
Policy control Excellent: per-neighbour filters, communities, local-pref Limited Limited
Failure blast radius Limited by ASN design, path hiding Link-state flood reaches the whole area Flood reaches the whole level
Operational familiarity Needs ASN plan Familiar to most teams Less familiar, no IP dependency
Overlay fit One protocol for underlay and EVPN Needs BGP for EVPN anyway Needs BGP for EVPN anyway

eBGP underlay

RFC 7938 documents the design used in very large data centres: eBGP on every link, with one routing protocol across the fabric. Common ASN schemes:

  • One ASN per leaf, one ASN for all spines. Simple, and loop prevention works naturally. A leaf rejects routes containing its own ASN, which prevents path hunting through other leaves.
  • One ASN per leaf pair (for vPC or MLAG) and one ASN per spine tier. Uses fewer ASNs.
  • Same ASN on all leaves needs allowas-in or equivalent and weakens loop prevention. Avoid it unless you have a reason.

Because leaves have different ASNs, two paths through different spines have the same AS-path length but different AS numbers. Many implementations require a setting to treat them as equal for multipath, such as bestpath as-path multipath-relax on NX-OS. Without it you will see one best path instead of ECMP.

Use /31 point-to-point addresses (RFC 3021) or unnumbered interfaces if your platform supports them for BGP, to save address space and automate cleanly.

OSPF and IS-IS

An IGP underlay is fine for small and medium fabrics and is often easier to troubleshoot. Rules:

  • Use point-to-point network type on all fabric links so there is no DR election.
  • Run a single area for a single pod. Do not stretch one area over multiple pods if you plan to grow.
  • Tune SPF throttle and enable BFD.
  • IS-IS needs no IP addressing on links to form adjacencies, which some operators like for IPv6 or unnumbered designs.

EVPN still needs iBGP or eBGP for the overlay. With an IGP underlay, the common pattern is iBGP with the spines as route reflectors. That is two protocols to run, which is the trade-off against eBGP.

Tip

If you are starting a new fabric and expect more than one pod, pick eBGP underlay with a documented ASN scheme. It is easier to apply policy at the pod boundary later.

Border and services leaves

Not every leaf connects servers. Dedicated leaves keep north-south and service traffic away from compute racks.

  • Border leaf: connects the fabric to the outside: WAN, internet edge, campus, other data centres. It exchanges routes with external routers, often via VRF-lite or EVPN type-5 routes, and carries traffic for the whole fabric. Deploy in pairs, with enough bandwidth to the spines (often 1:1) and diverse external uplinks.
  • Services leaf: hosts firewalls, load balancers or other appliances. Stateful devices need symmetric paths, so design for it explicitly with active/standby pairs or clusters, and test failover with live flows.
  • Storage or compute-specific leaves: group hosts with special requirements, such as lossless Ethernet for storage, at a different oversubscription.
graph TB
    SP["Spines"] --- CL["Compute leaves"]
    SP --- SL["Services leaves"]
    SP --- BL["Border leaves"]
    BL --- WAN["WAN / Internet / DCI"]
    SL --- FW["Firewalls and load balancers"]
    CL --- SRV["Servers"]

Do not attach border or services functions to the spine. It makes the spine stateful or policy-heavy, grows its failure domain, and breaks the property that spines are interchangeable.

Scaling: multi-pod and super-spine

A two-tier fabric is limited by spine port count: one leaf per spine port. When you run out, there are two common options.

Add a super-spine tier (five-stage Clos). Group leaves and spines into pods. Each pod is a self-contained two-tier fabric. Spines in every pod uplink to a super-spine layer. Inter-pod traffic takes leaf, spine, super-spine, spine, leaf (four hops). Intra-pod traffic is unchanged. Size the pod-to-super-spine bandwidth to the expected inter-pod traffic. Placing workloads that talk a lot in the same pod keeps that link lightly loaded.

Multi-pod or multi-site with overlay interconnect. Pods run their own underlay and are connected through border or gateway nodes using eBGP EVPN between sites. This gives stronger isolation at the cost of gateway complexity, and it is the usual choice when pods are in different buildings or cities.

graph TB
    SS1["Super-spine 1"] --- P1S["Pod 1 spines"]
    SS1 --- P2S["Pod 2 spines"]
    SS2["Super-spine 2"] --- P1S
    SS2 --- P2S
    P1S --- P1L["Pod 1 leaves"]
    P2S --- P2L["Pod 2 leaves"]

At this size, watch for hash polarisation (see above), ASN scheme exhaustion, and the number of BGP sessions on the super-spine. A scheme where all spines in a pod share an ASN, and each pod has its own, keeps AS-path lengths consistent.

Failure domains and maintenance

A failure domain is the set of things that break together. Design them small.

  • Single link or leaf: affects one rack or one uplink. Dual-homed servers (vPC/MLAG or EVPN multihoming) survive a leaf loss.
  • Single spine: removes 1/N of the fabric capacity but no reachability. With 4 spines you lose 25 percent. With 2 spines you lose half, which is why many designs start at 4.
  • Control plane: a bad config pushed everywhere can take down the entire fabric. Roll changes out per pod, or per tier, and keep the underlay config template separate from tenant and overlay changes.
  • Pod: a fault or maintenance in one pod should not affect another. Use route filters at pod boundaries and limit what you redistribute.
  • Shared services: DHCP, DNS, NTP, controllers and out-of-band management should not depend on a single leaf pair.

Maintenance practice

  1. Drain before you touch. Use a maintenance mode or raise route metrics so traffic leaves the device before you shut it down. With BGP, a common approach is to attach a community or prepend AS-path so neighbours prefer other paths, then wait for traffic to drop.
  2. One spine at a time. Confirm with interface counters that the other spines take the load without exceeding utilisation targets. This is where the failure-state oversubscription math matters.
  3. Use BFD. BFD on fabric links turns failure detection from seconds to well under a second, which helps in-flight traffic during an unplanned loss.
  4. Check before and after. Compare BGP neighbour counts, route counts and ECMP path counts against a baseline. A missing path is easy to miss because traffic still flows on the rest.
  5. Plan uplink and spine upgrades as capacity events. Adding spines changes ECMP group sizes everywhere, which causes re-hashing. Do it in a window.

Warning

A leaf that loses all uplinks but keeps its server links up will blackhole traffic from dual-homed hosts unless it withdraws them. Enable the vendor feature that shuts server-facing ports when uplinks are lost (often called uplink tracking or fabric-link tracking).

Common mistakes

  • Designing only for the healthy state. Oversubscription that looks fine with all spines up can congest with one down.
  • Too few spines. Two spines means 50 percent capacity loss on any failure and a coarse ECMP group.
  • Mixed link speeds in one ECMP group. Flows balance by count, not capacity.
  • Forgetting to raise maximum paths or enable multipath relaxation, so the fabric runs on one path.
  • IP-only hashing or a fixed VXLAN source port, which concentrates traffic.
  • Same ASN everywhere with loop-prevention disabled, which can create path hunting and routing loops during failures.
  • Putting firewalls or routers on the spine or cabling leaf to leaf outside an MLAG pair.
  • No uplink tracking on dual-homed leaves.
  • Sizing the border at the same ratio as compute leaves. The border concentrates everything.
  • Ignoring growth. Reserve spine ports and plan the super-spine move before you need it; recabling a live fabric is expensive.

If you want to practise the routing side of this, the CCNP course has a lab setup guide and a study plan, and the course home covers BGP and OSPF fundamentals.

References