Skip to content
Log in Sign up My account

Packet Walk: How a Packet Crosses a Spine-Leaf EVPN VXLAN Fabric

This article follows real traffic through the fabric built in the EVPN VXLAN lab. It covers how a leaf learns a host, which BGP route carries what, how ARP suppression answers a request without flooding, what the VXLAN packet looks like on the wire, and why routed traffic between subnets crosses two routing hops (symmetric IRB). Read it with the lab open so you can run the show commands as you go.

The cast

graph TB
    S1["Spine1"]
    S2["Spine2"]
    L1["Leaf1<br/>VTEP 10.255.1.11"]
    L2["Leaf2<br/>VTEP 10.255.1.12"]
    H1["H1 10.10.10.11<br/>VLAN 10"]
    H2["H2 10.10.10.12<br/>VLAN 10"]
    H3["H3 10.10.20.13<br/>VLAN 20"]
    S1 --- L1
    S1 --- L2
    S2 --- L1
    S2 --- L2
    L1 --- H1
    L2 --- H2
    L2 --- H3
  • L2 VNI 10010 is VLAN 10, L2 VNI 10020 is VLAN 20, L3 VNI 50001 is VRF TENANT1.
  • Every leaf holds the anycast gateway 10.10.10.1 and 10.10.20.1 with MAC 0000.2222.3333.
  • The spines are IP routers and BGP route reflectors. They never look at a VNI.

Part 1: What the control plane knows before the first data packet

EVPN replaces flood-and-learn with BGP advertisements. Three route types matter here.

Route type Name Carries Used for
2 MAC/IP advertisement Host MAC, optionally host IP, L2 VNI, optionally L3 VNI and router MAC Bridging and host routing
3 Inclusive multicast VTEP IP, L2 VNI Building the flood list for BUM traffic
5 IP prefix A prefix, L3 VNI, router MAC Routing to subnets and external prefixes

Learning a host

When H2 first sends a frame into Leaf2, the leaf learns its MAC on the access port in the normal way. Leaf2 puts that MAC in the local table for VLAN 10 and, because the VLAN is bound to VNI 10010, advertises it:

sequenceDiagram
    participant H2
    participant L2 as Leaf2
    participant RR as Spines RR
    participant L1 as Leaf1
    H2->>L2: Frame or ARP from H2
    L2->>L2: Learn MAC on Eth1/3, VLAN 10
    L2->>RR: BGP EVPN type 2 for H2 MAC
    RR->>L1: Reflect type 2, next hop 10.255.1.12
    L1->>L1: Install MAC, VNI 10010, via VTEP 10.255.1.12

When Leaf2 also learns H2's IP, from ARP or from the host's gratuitous ARP, it advertises a second type 2 that includes the IP. This one has two labels: the L2 VNI 10010 for bridging, and the L3 VNI 50001 for routing. It also carries the route target for each and a router MAC extended community holding Leaf2's own router MAC. The IP-bearing route is what lets Leaf1 route to H2 as a /32 host route in VRF TENANT1.

Leaf2 also sent a type 3 route earlier, as soon as VNI 10010 came up. Leaf1 uses it to learn that 10.255.1.12 wants BUM traffic for VNI 10010.

Check it:

Leaf1# show bgp l2vpn evpn route-type 2
Leaf1# show l2route evpn mac-ip all

Part 2: Same subnet, H1 to H2 (L2 VNI)

H1 (10.10.10.11) wants to reach H2 (10.10.10.12). H1 does not know H2's MAC.

Step 1: ARP request and suppression

H1 broadcasts an ARP request. In a classical VXLAN network this would flood across every VTEP. With ARP suppression enabled on VNI 10010 the leaf behaves differently.

flowchart TB
    A["H1 sends ARP request<br/>who has 10.10.10.12"] --> B{"Leaf1 ARP suppression cache<br/>has 10.10.10.12?"}
    B -->|"Yes"| C["Leaf1 replies to H1 locally<br/>with H2 MAC, request not flooded"]
    B -->|"No"| D["Leaf1 floods via ingress replication<br/>to VTEPs from type 3 routes"]
    D --> E["H2 answers, Leaf2 learns H2<br/>and advertises a type 2 with MAC and IP"]

The suppression cache is built from type 2 routes carrying IPs. When Leaf1 has an entry for 10.10.10.12, it answers H1 itself, using H2's real MAC. The request never leaves the leaf. This is the main reason EVPN fabrics keep ARP noise low. A silent host, one that has never sent anything, is not in the cache, so its first ARP is flooded. After it replies, every leaf knows it.

Leaf1# show ip arp suppression-cache detail

Step 2: The data frame

H1 now sends a unicast Ethernet frame with destination MAC equal to H2's MAC. Leaf1 looks up that MAC in the table for VLAN 10 and finds a remote entry: VNI 10010, next hop VTEP 10.255.1.12. It encapsulates.

Original frame from H1
+--------------------+--------------------+-----------------+
| Dst MAC = H2       | Src MAC = H1       | IP / payload    |
+--------------------+--------------------+-----------------+

After Leaf1 encapsulation
+---------------+---------------+---------------+-----------+-------------+------------------+
| Outer Eth     | Outer IP      | Outer UDP     | VXLAN     | Inner frame |
| dst: Spine MAC| src 10.255.1.11| dst 4789     | VNI 10010 | (H1 to H2)  |
| src: Leaf1 MAC| dst 10.255.1.12| src: hash    | flag I=1  |             |
+---------------+---------------+---------------+-----------+-------------+

Details that are worth knowing:

  • The overhead is 50 bytes (14 outer Ethernet, 20 outer IPv4, 8 UDP, 8 VXLAN), plus 4 if the underlay link carries a VLAN tag. That is why the underlay MTU is 9216 in the lab.
  • UDP destination port is 4789, the IANA port for VXLAN. The source port is derived from a hash of the inner headers. Spines use it for ECMP, so one flow stays on one path and different flows spread across both spines.
  • The VXLAN header holds a 24-bit VNI, so up to about 16 million segments.

Step 3: Across the underlay

Leaf1 has two equal-cost routes to 10.255.1.12, via Spine1 and via Spine2. It picks one by hash. The spine sees an ordinary IP packet to 10.255.1.12, routes it with its OSPF table, and decrements the outer TTL. It does not know about VNIs, MACs or hosts.

Step 4: Decapsulation

Leaf2 receives a packet addressed to its own VTEP address, UDP 4789. It strips the outer headers, reads VNI 10010 and maps it to VLAN 10. It looks up the inner destination MAC in that VLAN, finds H2 on a local port, and forwards the original frame. H2 sees a frame as if both hosts were on one switch.

Part 3: Different subnets, H1 to H3 (symmetric IRB)

H1 (VLAN 10) wants to reach H3 (10.10.20.13, VLAN 20). They are in different subnets, so H1 sends to its default gateway.

Step 1: Gateway ARP

H1 ARPs for 10.10.10.1. Leaf1 answers locally with the anycast MAC 0000.2222.3333. This is a normal local reply, not a fabric event. Every leaf would answer the same way.

Step 2: Ingress leaf routes

H1 sends a frame with destination MAC 0000.2222.3333 and destination IP 10.10.20.13. Leaf1 sees its own gateway MAC, so it does a Layer 3 lookup in VRF TENANT1 for 10.10.20.13. It finds the /32 host route that came from the type 2 route (MAC+IP) that Leaf2 sent. The route says: next hop VTEP 10.255.1.12, L3 VNI 50001, router MAC of Leaf2.

Leaf1 then decrements the inner TTL and rewrites the inner Ethernet header. The source MAC becomes Leaf1's router MAC and the destination MAC becomes Leaf2's router MAC, taken from the extended community. This is the key move of symmetric IRB: Leaf1 does not need to know VLAN 20 or H3's MAC. It addresses the packet to the remote leaf.

Inner frame as sent into the tunnel
+----------------------+----------------------+--------------------------+
| Dst MAC = Leaf2 rMAC | Src MAC = Leaf1 rMAC | IP src 10.10.10.11       |
|                      |                      | IP dst 10.10.20.13, TTL-1|
+----------------------+----------------------+--------------------------+
VXLAN header: VNI 50001 (the L3 VNI, not 10010 or 10020)
Outer IP: 10.255.1.11 to 10.255.1.12, UDP 4789

Step 3: Underlay and decap

The spine forwards it exactly as before. Leaf2 decapsulates, reads VNI 50001, and maps it to VRF TENANT1 instead of a VLAN.

Step 4: Egress leaf routes

Leaf2 looks up 10.10.20.13 in TENANT1. This is a connected host on interface Vlan20. Leaf2 decrements TTL again, finds H3's MAC in the ARP table, rewrites the Ethernet header (source: anycast gateway MAC for VLAN 20, destination: H3's MAC), and sends the frame out the access port in VLAN 20.

sequenceDiagram
    participant H1
    participant L1 as Leaf1
    participant SP as Spine
    participant L2 as Leaf2
    participant H3
    H1->>L1: dst MAC anycast GW, dst IP 10.10.20.13
    L1->>L1: Route in TENANT1, TTL-1, rewrite MACs to router MACs
    L1->>SP: VXLAN VNI 50001, outer 10.255.1.11 to 10.255.1.12
    SP->>L2: Route on outer IP, outer TTL-1
    L2->>L2: Decap, VNI 50001 maps to TENANT1, route, TTL-1
    L2->>H3: Frame on VLAN 20, dst MAC H3

The inner TTL is reduced twice, once by each leaf. In a traceroute from H1 to H3 you see the Leaf1 gateway and then the Leaf2 gateway as hops, even though the spine, which only touches the outer packet, is not shown.

Why symmetric

In asymmetric IRB the ingress leaf routes and bridges, so it must carry every VLAN of every destination, and the return path takes a different route. In symmetric IRB both directions use the same L3 VNI and each leaf only needs the VLANs that are local to it. The cost is the second routing hop and the L3 VNI setup. This scales better, so it is the standard design.

Return traffic works the same way mirrored: Leaf2 routes, encapsulates into VNI 50001, and Leaf1 routes into VLAN 10.

Part 4: Prefix routes and silent hosts

Type 2 routes only exist for hosts the fabric has seen. To reach a subnet or an external prefix, leaves advertise type 5 routes. A border leaf that knows a default route or a WAN prefix in VRF TENANT1 advertises it as a type 5 carrying the L3 VNI and its router MAC. A remote leaf uses that route to forward to the border leaf, again over VNI 50001.

For a host that has never spoken, the ingress leaf has no /32. Depending on the platform and configuration, it routes using the connected subnet route to the SVI, which triggers an ARP (glean) on that VLAN. The ARP floods via the type 3 routes, the host answers, and from then on the host is known. Some designs add a "silent host" feature to speed this up.

Part 5: Host moves

When a host moves between leaves, the new leaf advertises the same MAC with a higher MAC mobility sequence number in an extended community. All leaves prefer the highest sequence number and update their tables. The old leaf withdraws its route. That is how a VM can migrate without changing its gateway.

Commands to watch it happen

Leaf1# show bgp l2vpn evpn route-type 2
Leaf1# show bgp l2vpn evpn route-type 3
Leaf1# show l2route evpn mac all
Leaf1# show ip route vrf TENANT1 10.10.20.13
Leaf1# show nve peers
Leaf1# show mac address-table vlan 10
Leaf1# show ip arp suppression-cache detail

Capture on a spine link (in a virtual lab, on the link between nodes) and open the file in a packet analyser. The VXLAN dissector decodes UDP 4789, shows the VNI, and lets you compare an L2 packet with VNI 10010 against a routed one with VNI 50001. Seeing the router MACs in the inner Ethernet header of the routed packet makes symmetric IRB concrete.

For the exam context see the CCNP home page and the study plan.

References