Skip to content
Log in Sign up My account

4.6 and 4.7 Troubleshooting NetFlow and Catalyst Center Assurance

Blueprint 4.6 and 4.7

  • 4.6 Troubleshoot NetFlow (v5, v9, Flexible NetFlow)
  • 4.7 Troubleshoot network problems using Cisco DNA Center assurance (now Catalyst Center): connectivity, monitoring, device health, network health

NetFlow questions show a Flexible NetFlow configuration and a symptom (empty cache, collector receives nothing, collector cannot decode). Assurance questions describe a problem and ask which Catalyst Center tool or score finds it fastest.

NetFlow versions

A flow is a set of packets that share the same key fields. The router keeps a cache of active flows, counts packets and bytes per flow, and exports expired flows to a collector over UDP.

NetFlow v5 NetFlow v9 IPFIX (v10)
Record format Fixed Template based Template based, IETF standard
Key fields Fixed seven: source IP, destination IP, source port, destination port, protocol, ToS, input interface Defined by the template Defined by the template
IPv6, MPLS, custom fields No Yes Yes, plus variable-length fields
Collector needs templates first No Yes Yes
Standard Cisco Cisco, RFC 3954 (informational) RFC 7011
Configured with Traditional NetFlow Traditional or Flexible NetFlow Flexible NetFlow 
sequenceDiagram
  participant R as R1 exporter
  participant C as Collector UDP 2055
  R->>C: Template FlowSet, ID 256: fields and lengths
  R->>C: Data FlowSet, ID 256, flow records
  Note over C: Collector restarts and loses templates
  R->>C: Data FlowSet, ID 256
  Note over C: Cannot decode, records discarded
  R->>C: Template resent after template timeout
  R->>C: Data FlowSet, decoded again

Exam trap

After a collector restart, v9 and IPFIX data cannot be decoded until the exporter resends its templates. If a question says "the collector receives packets but shows no flows for several minutes after a restart", lower the template timeout on the exporter (template data timeout 60) rather than looking for a routing fault.

Traditional NetFlow (IOS)

Classic IOS images such as IOSv still accept the traditional commands. IOS XE routing platforms use Flexible NetFlow only, so you will mostly meet these in older exhibits.

interface GigabitEthernet0/0
 ip flow ingress
 ip flow egress
ip flow-export version 9
ip flow-export destination 192.168.100.50 2055
ip flow-export source Loopback0
!
R3# show ip cache flow
R3# show ip flow export

Flexible NetFlow building blocks

graph LR
  REC["flow record<br/>match = key fields<br/>collect = non-key fields"] --> MON["flow monitor<br/>record, exporter, cache timeouts"]
  EXP["flow exporter<br/>destination, source, port, protocol"] --> MON
  SAM["sampler<br/>optional, 1 out-of N"] -.-> INT
  MON --> INT["interface<br/>ip flow monitor NAME input or output"]
flow record FR-IPV4
 match ipv4 source address
 match ipv4 destination address
 match ipv4 protocol
 match transport source-port
 match transport destination-port
 match ipv4 tos
 match interface input
 collect interface output
 collect counter bytes long
 collect counter packets long
 collect timestamp absolute first
 collect timestamp absolute last
!
flow exporter FE-NMS
 destination 192.168.100.50
 source Loopback0
 transport udp 2055
 export-protocol netflow-v9
 template data timeout 60
!
flow monitor FM-LAN
 record FR-IPV4
 exporter FE-NMS
 cache timeout active 60
 cache timeout inactive 15
!
interface GigabitEthernet2
 ip flow monitor FM-LAN input
Element What goes wrong
match lines They define a flow. Too few keys merge different conversations; too many create a huge cache.
collect lines Counters and timestamps. Missing collect counter bytes means the collector shows flows with no volume.
Exporter destination and transport Wrong address or port: the router exports happily, the collector hears nothing. If transport udp is omitted, IOS uses its default port (9995), which many collectors do not listen on.
Exporter source An interface with no route back, or an address the collector does not expect
Monitor exporter Without it, the cache fills locally and nothing is exported
Cache timeouts Default active 1800 seconds, inactive 15 seconds. With the default active timeout, a long download is exported only every 30 minutes.
Interface direction input sees traffic arriving on that interface. Monitoring only input on the LAN interface misses the return traffic arriving from the WAN.

Predefined records save typing: record netflow ipv4 original-input reproduces the traditional key fields.

Sampling

sampler SAMPLE-100
 mode random 1 out-of 100
interface GigabitEthernet1
 ip flow monitor FM-LAN sampler SAMPLE-100 input

Sampling reduces CPU and cache use on fast links. The collector must know the sampling rate to scale the counters back up; otherwise reported volumes look 100 times too small.

Verifying and troubleshooting Flexible NetFlow

Work from the interface outwards: is the monitor attached, is the cache filling, is the exporter sending, is the collector receiving.

R1# show flow interface GigabitEthernet2
Interface GigabitEthernet2
  FNF:  monitor:         FM-LAN
        direction:       Input
        traffic(ip):     on

R1# show flow monitor FM-LAN cache format table
  Cache type:                               Normal (Platform cache)
  Cache size:                                10000
  Current entries:                              14
  Flows added:                                 312
  Flows aged:                                  298
    - Active timeout      (    60 secs)        201
    - Inactive timeout    (    15 secs)         97

IPV4 SRC ADDR    IPV4 DST ADDR    TRNS SRC PORT  TRNS DST PORT  IP PROT  bytes long  pkts long
10.10.10.21      192.168.100.50          51734            443        6       84213        122
...

R1# show flow exporter FE-NMS statistics
Flow Exporter FE-NMS:
  Packet send statistics (last cleared 00:22:10 ago):
    Successfully sent:         96                    (118656 bytes)
  Client send statistics:
    Client: Flow Monitor FM-LAN
      Records added:           298
        - sent:                298

(Output trimmed. Field names vary slightly by release.)

Other useful commands: show flow exporter FE-NMS (destination, source, port, protocol), show flow exporter FE-NMS templates, show flow monitor FM-LAN (record, exporter, timeouts), show flow record FR-IPV4.

Symptom What it points to
show flow interface shows nothing Monitor not applied to that interface
Cache empty Wrong direction, wrong interface, or no matching traffic (an IPv4 record does not see IPv6)
Cache fills, exporter sent counter at 0 Exporter not attached to the monitor, or export failing (no route to destination, source interface down)
Exporter sending, collector silent Wrong port or address, ACL or firewall blocking UDP, collector bound to another port
Collector receives but cannot decode Templates not yet received (v9/IPFIX); lower template data timeout
Long flows appear late or as huge spikes Active timeout at the 1800-second default

Exam trap

IOS refuses to change the record of a flow monitor while that monitor is applied to an interface. The fix sequence is: remove ip flow monitor from the interfaces, change the record, reapply. A question that shows the change being rejected is testing this.

Exam trap

The cache lists active flows but the collector receives nothing, and the exporter shows the correct destination. Compare the exporter's transport udp port with the port the collector listens on. That mismatch is the most common exhibit fault.

Catalyst Center Assurance

Catalyst Center (formerly DNA Center) Assurance collects telemetry from managed devices, correlates it, and presents health scores, issues and troubleshooting tools. The blueprint still says Cisco DNA Center; the exam may use either name.

graph LR
  DEV["Managed devices"] -->|"SNMP, syslog, NetFlow,<br/>model-driven telemetry"| CC["Catalyst Center<br/>Assurance"]
  CC --> NH["Network health<br/>% of healthy devices"]
  CC --> DH["Device 360<br/>KPIs, issues, history"]
  CC --> CH["Client health and Client 360"]
  CC --> IS["Issues with<br/>suggested actions"]
  CC --> PT["Path trace"]

Health scores

Score Range Colour
Poor 1 to 3 Red
Fair 4 to 7 Orange
Good 8 to 10 Green
No data none Grey
  • Device health is based on KPIs such as reachability, CPU, memory, interface errors and utilisation, and for wireless devices radio metrics. A device's score is driven down by its worst KPI.
  • Network health is the percentage of network devices that are healthy (score 8 to 10), shown overall and per category (core, distribution, access, router, wireless).
  • Client health is the percentage of clients that are healthy, split into wired and wireless.

A drop in network health tells you something is wrong somewhere. The drill-down tells you where: the health dashboard, then the device category, then the device in Device 360, which shows its score history, KPIs, issues, neighbour topology and events.

Tools and when to use them

Problem Assurance tool
"Which devices are unhealthy right now?" Network Health dashboard
"What was wrong with this switch at 02:00 last night?" Device 360 with the timeline (time travel)
"This user cannot reach the ERP server" Client 360, then Path Trace between the client and server
"Is an ACL dropping this flow?" Path Trace with the ACL trace option
"Run the same show command on 40 switches" Command Runner
"What should I do about this alert?" Issues: each issue has priority (P1 to P4), impact and suggested actions

Path Trace takes a source and destination (and optionally ports and protocol) and builds the hop-by-hop path from the forwarding information Catalyst Center collects from devices. It can overlay interface statistics, QoS and ACL analysis, marking the hop where an ACL would block the flow. Both endpoints and the devices on the path must be known to Catalyst Center.

Why Assurance shows no data

Assurance can only report on what it receives. When a device shows grey (no data) or wrong values, troubleshoot the telemetry path, much as you would troubleshoot SNMP or NetFlow:

Check Why
Device is in inventory and its status is managed and reachable Discovery, CLI and SNMP credentials must work, otherwise collection fails
Telemetry settings pushed to the device Catalyst Center must be configured as the device's syslog server, SNMP trap receiver and NetFlow collector, and model-driven telemetry subscriptions must be in place (Network Settings > Telemetry in the GUI)
Reachability from the device to Catalyst Center Syslog, traps and NetFlow are sent by the device; an ACL or missing route blocks them while polling might still work
NTP on devices and Catalyst Center Events are correlated by time; unsynchronised clocks produce wrong timelines
Device software supported Some KPIs need a minimum IOS XE release

Exam trap

A device appears in the inventory and Catalyst Center can poll it, but its Assurance pages have no syslog-based issues and no flow data. Polling uses SNMP and CLI from Catalyst Center to the device; syslog, traps and NetFlow go the other way and need the device configured to send them. Push the telemetry settings to the device, and check the path back to Catalyst Center.

Exam trap

Network health is a percentage of healthy devices, not an average of their scores. One failing core switch in a network of 100 healthy access switches barely moves the network health number but shows clearly as a Poor device in its category. Read the per-category numbers.

Lab: Flexible NetFlow break/fix

Topology: module lab. R1 (Catalyst 8000v) exports to the NMS. On the NMS install nfdump and run nfcapd -w -D -l /tmp/nf -p 2055, or run sudo tcpdump -n udp port 2055 if you only want to see packets.

  1. Configure the FR-IPV4, FE-NMS and FM-LAN blocks above on R1 and apply the monitor to Gi2 input and Gi1 input (so both directions of HOST1 traffic are seen).
  2. Generate traffic: from HOST1 ping the NMS with a repeat count of 1000, and Telnet or SSH to it.
  3. Verify with show flow interface, show flow monitor FM-LAN cache format table and show flow exporter FE-NMS statistics. On the NMS, nfdump -R /tmp/nf should list the flows after a minute.
  4. Save the baseline.
  5. Fault A. Under the exporter set transport udp 9996. The router sent counter rises; the collector receives nothing. Find it with show flow exporter FE-NMS and tcpdump. Restore 2055.
  6. Fault B. Remove exporter FE-NMS from the monitor. The cache fills, nothing is exported. Restore it.
  7. Fault C. Try to remove collect counter packets long from FR-IPV4 while the monitor is applied. Note the rejection, then do it properly: remove the monitor from both interfaces, change the record, reapply.
  8. Fault D. Restart nfcapd on the NMS with the exporter's template data timeout set back to 600. Note how long it takes for flows to decode again, then set it to 60.
  9. Fault E. Remove the monitor from Gi1 only. Look at the cache and note that only the HOST1-to-NMS direction is recorded.

Expected verification output (baseline):

R1# show flow exporter FE-NMS
Flow Exporter FE-NMS:
  Description:              User defined
  Export protocol:          NetFlow Version 9
  Transport Configuration:
    Destination IP address: 192.168.100.50
    Source IP address:      10.255.255.1
    Source Interface:       Loopback0
    Transport Protocol:     UDP
    Destination Port:       2055

For Assurance, practise on the Cisco DevNet always-on Catalyst Center sandbox (search "Catalyst Center sandbox" on developer.cisco.com). Open the Health dashboard, drill into a device with a low score, read its Device 360 KPIs, and run a Path Trace between two hosts.

Quick check

A Flexible NetFlow monitor is applied with ip flow monitor FM input on the LAN interface only. Users complain the collector shows uploads but not downloads. Why?

input on the LAN interface records only traffic arriving from the LAN, which is the upload direction. Downloads arrive on the WAN interface. Apply the monitor input on the WAN interface too (or output on the LAN interface).

Which NetFlow export format cannot carry IPv6 flows, and why?

NetFlow v5. Its record format is fixed and only has fields for IPv4 addresses. v9 and IPFIX use templates and can describe IPv6, MPLS and other fields.

A help desk ticket says one user cannot reach an application server. Which Catalyst Center Assurance tool shows the hop-by-hop path and whether an ACL blocks it?

Path Trace, run from Client 360 or the Path Trace tool, with the ACL trace option enabled.

References

  • Cisco IOS XE Flexible NetFlow Configuration Guide: "Getting Started with Flexible NetFlow" and "Configuring Data Export for Flexible NetFlow" (search on cisco.com)
  • RFC 3954 (NetFlow v9): rfc-editor.org/rfc/rfc3954
  • RFC 7011 (IPFIX): rfc-editor.org/rfc/rfc7011
  • Cisco Catalyst Center Assurance User Guide: "Monitor and Troubleshoot the Health of Your Network" (search on cisco.com)
  • Cisco DevNet: Catalyst Center API documentation and sandboxes
  • Cisco Live On-Demand Library: search "Flexible NetFlow" and "Catalyst Center Assurance"