Skip to content
Log in Sign up My account

EVPN VXLAN Fabric Lab on Nexus 9000v

This lab builds a two-spine, two-leaf VXLAN EVPN fabric on NX-OS. You configure an OSPF underlay, iBGP EVPN with the spines as route reflectors, a Layer 2 VNI per VLAN, a Layer 3 VNI per tenant VRF, and a distributed anycast gateway. Then you verify the control plane and data plane with the commands that matter. Full configs are given for one spine and one leaf, with the differences for the others listed.

Topology and addressing

graph TB
    S1["Spine1<br/>lo0 10.255.0.1"]
    S2["Spine2<br/>lo0 10.255.0.2"]
    L1["Leaf1<br/>lo0 10.255.0.11<br/>lo1 VTEP 10.255.1.11"]
    L2["Leaf2<br/>lo0 10.255.0.12<br/>lo1 VTEP 10.255.1.12"]
    H1["H1 VLAN 10<br/>10.10.10.11"]
    H2["H2 VLAN 10<br/>10.10.10.12"]
    H3["H3 VLAN 20<br/>10.10.20.13"]
    S1 --- L1
    S1 --- L2
    S2 --- L1
    S2 --- L2
    L1 --- H1
    L2 --- H2
    L2 --- H3
Link Subnet
Spine1 - Leaf1 10.1.1.0/31 (Spine .0, Leaf .1)
Spine1 - Leaf2 10.1.1.2/31 (Spine .2, Leaf .3)
Spine2 - Leaf1 10.1.2.0/31 (Spine .0, Leaf .1)
Spine2 - Leaf2 10.1.2.2/31 (Spine .2, Leaf .3)

Design values used throughout:

Item Value
BGP AS 65000, iBGP, spines are route reflectors
VLAN 10 / L2 VNI 10010, subnet 10.10.10.0/24, gateway 10.10.10.1
VLAN 20 / L2 VNI 10020, subnet 10.10.20.0/24, gateway 10.10.20.1
VRF / L3 VNI TENANT1 / 50001, transit VLAN 500
Anycast gateway MAC 0000.2222.3333

Image notes: use a recent Nexus 9000v image and read its release notes for which VXLAN features are supported on the virtual platform. The 9000v is for control plane and feature learning, and its forwarding behaviour can differ from hardware. Each node wants several GB of RAM, so size your host first (see the platform comparison).

Step 1: Underlay

The underlay has one job: give every VTEP loopback IP reachability, and carry jumbo frames. VXLAN adds 50 bytes, so set the fabric links to a large MTU. The spines and leaves use point-to-point /31 links with OSPF.

! Spine1
feature ospf
feature bgp
nv overlay evpn

hostname Spine1
interface loopback0
  ip address 10.255.0.1/32
  ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
  router-id 10.255.0.1

interface Ethernet1/1
  description to Leaf1
  no switchport
  mtu 9216
  ip address 10.1.1.0/31
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown
interface Ethernet1/2
  description to Leaf2
  no switchport
  mtu 9216
  ip address 10.1.1.2/31
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

Spine2 is the same with loopback 10.255.0.2, router ID 10.255.0.2 and links 10.1.2.0/31 and 10.1.2.2/31.

! Leaf1
feature ospf
feature bgp
nv overlay evpn

hostname Leaf1
interface loopback0
  description router-id and BGP peering
  ip address 10.255.0.11/32
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  description VTEP source
  ip address 10.255.1.11/32
  ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY
  router-id 10.255.0.11

interface Ethernet1/1
  description to Spine1
  no switchport
  mtu 9216
  ip address 10.1.1.1/31
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown
interface Ethernet1/2
  description to Spine2
  no switchport
  mtu 9216
  ip address 10.1.2.1/31
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

Leaf2 uses loopback0 10.255.0.12, loopback1 10.255.1.12, and addresses 10.1.1.3/31 and 10.1.2.3/31 on its uplinks.

Verify the underlay before anything else:

Leaf1# show ip ospf neighbors
Leaf1# show ip route ospf
Leaf1# ping 10.255.1.12 source 10.255.1.11

Both spines should be FULL neighbours, and 10.255.1.12 should have two equal-cost paths. If the ping works, the underlay is done.

Step 2: BGP EVPN overlay

The spines are route reflectors for the l2vpn evpn address family. Leaves peer with both spines by loopback. Note that the spines do not need to know about VNIs at all.

! Spine1
router bgp 65000
  router-id 10.255.0.1
  neighbor 10.255.0.11
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client
  neighbor 10.255.0.12
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client
  address-family l2vpn evpn
    retain route-target all

retain route-target all is needed on a route reflector that has no local VRFs. By default NX-OS discards EVPN routes whose route targets it does not import, and the reflector would drop everything.

! Leaf1
router bgp 65000
  router-id 10.255.0.11
  neighbor 10.255.0.1
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.0.2
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended

Leaf2 is identical apart from the router ID 10.255.0.12. Extended communities carry the route targets and the router MAC, so they must be sent.

Spine1# show bgp l2vpn evpn summary

Both leaf neighbours should show as established. No routes yet, because no VNIs exist.

Step 3: Layer 2 VNIs

Enable the features, set the anycast gateway MAC, and map VLANs to VNIs. This goes on both leaves.

feature vn-segment-vlan-based
feature nv overlay
feature interface-vlan
feature fabric forwarding
fabric forwarding anycast-gateway-mac 0000.2222.3333

vlan 10
  name WEB
  vn-segment 10010
vlan 20
  name APP
  vn-segment 10020

Create the NVE interface (the VTEP) and attach the L2 VNIs. BGP ingress replication handles broadcast, unknown unicast and multicast without needing PIM in the underlay.

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    suppress-arp
    ingress-replication protocol bgp
  member vni 10020
    suppress-arp
    ingress-replication protocol bgp

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto

Repeat the vni stanza for the second VNI.

evpn
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto

Put the hosts in their VLANs on the access ports:

! Leaf1
interface Ethernet1/3
  switchport
  switchport access vlan 10
  no shutdown
! Leaf2
interface Ethernet1/3
  switchport
  switchport access vlan 10
  no shutdown
interface Ethernet1/4
  switchport
  switchport access vlan 20
  no shutdown

Warning

On physical Nexus 9000 switches, ARP suppression needs a TCAM carve for the ARP-ETHER region, set with hardware access-list tcam region arp-ether 256 double-wide and a reload. The virtual platform may not need or honour it. Check the configuration guide for your exact model and release.

Step 4: Layer 3 VNI and anycast gateway

For routing between VLANs, create the tenant VRF with an L3 VNI. A transit VLAN carries the L3 VNI mapping and gets an SVI with ip forward.

vlan 500
  name TENANT1-L3VNI
  vn-segment 50001

vrf context TENANT1
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan500
  no shutdown
  vrf member TENANT1
  ip forward
  no autostate

interface Vlan10
  no shutdown
  vrf member TENANT1
  ip address 10.10.10.1/24
  fabric forwarding mode anycast-gateway

interface Vlan20
  no shutdown
  vrf member TENANT1
  ip address 10.10.20.1/24
  fabric forwarding mode anycast-gateway

interface nve1
  member vni 50001 associate-vrf

router bgp 65000
  vrf TENANT1
    address-family ipv4 unicast
      advertise l2vpn evpn

Apply the same on both leaves. The SVI IP addresses are identical on every leaf by design: this is the anycast gateway. Every leaf answers ARP for 10.10.10.1 with the same MAC 0000.2222.3333, so a host can move between leaves without changing its default gateway.

associate-vrf is what makes this a symmetric design. Traffic between subnets is routed on the ingress leaf, carried in VNI 50001, and routed again on the egress leaf. The packet walk follows one packet through exactly this.

Step 5: Verification

Give each host an IP in its subnet with the gateway set (10.10.10.1 or 10.10.20.1). Then ping between them to generate the ARP traffic that creates MAC/IP routes.

Control plane

Leaf1# show nve peers
Leaf1# show nve vni
Leaf1# show bgp l2vpn evpn summary
Leaf1# show bgp l2vpn evpn route-type 2
Leaf1# show bgp l2vpn evpn route-type 3
Leaf1# show l2route evpn mac all
Leaf1# show l2route evpn mac-ip all

What to look for:

  • show nve peers lists 10.255.1.12 in state Up, learned by CP (control plane).
  • show nve vni lists 10010, 10020 as L2 and 50001 as L3, each state Up, with the mapped VLAN or VRF.
  • Route-type 3 (inclusive multicast) entries for each L2 VNI from the remote VTEP. These build the ingress replication list.
  • Route-type 2 entries carrying the host MAC and, once the host has ARPed, the IP. The type 2 for an IP shows two labels, the L2 VNI and the L3 VNI, plus a router MAC extended community.
  • show l2route evpn mac all shows local MACs with a producer of Local and remote MACs as BGP with the next hop being the remote VTEP.

Layer 3

Leaf1# show ip route vrf TENANT1
Leaf1# show vrf TENANT1
Leaf1# show ip arp vrf TENANT1
Leaf1# show bgp l2vpn evpn 10.10.20.13

The remote host routes appear in the TENANT1 table as /32s reached through the remote VTEP, with the L3 VNI 50001 shown in the route detail and an encapsulation of VXLAN.

Data plane tests

H1$ ping 10.10.10.12      ! same VLAN across the fabric, L2 VNI
H1$ ping 10.10.20.13      ! different subnet, L3 VNI
Leaf1# show vxlan interface
Leaf1# show interface nve1

If the L2 ping works but the routed ping fails, check show nve vni for the L3 VNI state and make sure advertise l2vpn evpn is configured under the VRF. If everything looks right in the control plane but pings fail in a virtual lab, suspect the virtual platform's data plane and MTU. Try smaller packets with ping ... packet-size 1000.

Troubleshooting order

  1. Underlay reachability between VTEP loopbacks, and MTU.
  2. BGP EVPN sessions up on every leaf.
  3. NVE up, VNIs up, same VNI numbers on both ends.
  4. Route targets match (auto RTs derive from the AS number and VNI, so the AS must match).
  5. Type 2 and type 3 routes present on the remote leaf.
  6. Host ARP in the right VRF.
  7. Anycast gateway MAC set, and identical on all leaves.

For the broader exam context see the CCNP home page. To see why each of these routes exists, read the packet walk.

References