Skip to content
Log in Sign up My account

4.1 Diagnostic Tools: Debugs, Ping, Traceroute, SNMP and Syslog

Blueprint 4.1: Diagnose network problems using tools such as debugs, conditional debugs, traceroute, ping, SNMP, and syslog

  • Debugs and conditional debugs: limiting debug output safely on a production device
  • Ping and extended ping: reading the result characters, sourcing, sizing and the DF bit
  • Traceroute: how it works on IOS XE and what each result character means
  • SNMP v2c and v3: polling, traps and informs, security levels
  • Syslog: severity levels, destinations, timestamps and the message format

The verb is diagnose. Expect exhibits of command output and configuration where you must say what the tool is telling you, or why a monitoring tool is not working.

A troubleshooting order that works

Most failures are found faster if you move from cheap, safe tools to expensive ones. Debugs are last because they cost CPU.

graph LR
  A["Symptom reported"] --> B["ping: is it reachable?"]
  B --> C["traceroute: where does it stop?"]
  C --> D["show commands and logs"]
  D --> E["conditional debug on the suspect device"]
  E --> F["Fix, verify, undebug all"]
Tool Answers the question Cost and risk
ping Is there two-way reachability, and how long does a round trip take? Very low
traceroute Which hops does the path cross, and where does it break? Low
Syslog What did the device report, and when? Low, if the levels are sensible
SNMP What are the counters, states and trends over time? Low, polling is scheduled
debug What exactly is the process doing, packet by packet? High: can saturate the CPU

Ping and extended ping

ping sends ICMP echo requests and waits for echo replies. A success proves the path works in both directions, which is the point to remember when a ping fails: the problem may be on the return path.

Default IOS XE ping: five 100-byte echoes, two-second timeout, sourced from the exit interface.

Result characters

Character Meaning
! Echo reply received
. Timed out waiting for a reply
U ICMP destination unreachable received
Q Source quench (destination too busy)
M Could not fragment (DF bit set and packet larger than an MTU on the path)
? Unknown packet type
& Packet lifetime (TTL) exceeded

A first . followed by !!!! is normal: the first packet waited for ARP to resolve the next hop.

Extended ping

Type ping with no arguments for the interactive dialog, or put the options on one line. The options you need most:

R1# ping 10.255.255.3 source Loopback0 repeat 100 size 1500 df-bit
R1# ping vrf CUSTOMER-A 10.20.0.1
R1# ping 2001:db8:ffff::3 source Loopback0
Option Why you use it
source Tests the return route to the loopback or LAN, not just the link subnet
size and df-bit Finds MTU problems, for example on GRE or IPsec tunnels
repeat Exposes intermittent loss
vrf Pings inside a VRF routing table
ToS (extended dialog) Sends marked traffic to test a QoS policy 
R1# ping 10.255.255.3 source Loopback0 size 1500 df-bit
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.255.255.3, timeout is 2 seconds:
Packet sent with a source address of 10.255.255.1
Packet sent with the DF bit set
M.M.M
Success rate is 0 percent (0/5)

The M results mean a router on the path has a smaller MTU and returned "fragmentation needed". Lower the size until you get ! to find the path MTU.

Exam trap

A ping from R1 to a remote loopback succeeds, but the same ping with source Loopback0 fails. The forward path is fine; the remote router has no route back to R1's loopback. The fix is on the routing side (advertise R1's loopback), not on the link.

Traceroute

IOS XE traceroute sends UDP probes to high destination ports (starting at 33434) with TTL 1, 2, 3 and so on, three probes per hop, up to a maximum TTL of 30. Each router that decrements the TTL to zero returns ICMP time exceeded, which reveals its address. The destination returns ICMP port unreachable, which ends the trace. Windows tracert uses ICMP echo instead of UDP, so a firewall can treat the two differently.

graph LR
  R1["R1"] -->|"TTL 1"| R2["R2"]
  R2 -->|"ICMP time exceeded"| R1
  R1 -->|"TTL 2"| R3["R3"]
  R3 -->|"ICMP time exceeded"| R1
  R1 -->|"TTL 3, UDP 33434+"| D["Destination"]
  D -->|"ICMP port unreachable"| R1
Character Meaning
* Probe timed out
!H Host unreachable
!N Network unreachable
!P Protocol unreachable
!A Administratively prohibited (an ACL dropped the probe and said so)
? Unknown packet type 
R1# traceroute 10.255.255.3 source Loopback0 numeric
Type escape sequence to abort.
Tracing the route to 10.255.255.3
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.12.2 1 msec 1 msec 1 msec
  2 10.1.23.3 2 msec !A  *  !A

Hop 2 has an ACL that denies the probes and returns administratively prohibited. Repeated addresses that alternate between two hops (10.1.12.2, 10.1.23.3, 10.1.12.2...) mean a routing loop. A row of * * * followed by a working next hop just means that one router does not send ICMP time exceeded, which is often deliberate rate limiting.

Exam trap

Traceroute shows the forward path only. Asymmetric routing is common, and the return traffic may take a completely different path. Use numeric to avoid waiting for DNS lookups on every hop.

Syslog

Severity levels

Level Keyword Example
0 emergencies System unusable
1 alerts Immediate action needed
2 critical Memory allocation failure
3 errors Interface down (%LINK-3-UPDOWN)
4 warnings Configuration or resource warning
5 notifications Line protocol change (%LINEPROTO-5-UPDOWN), config change
6 informational ACL log hits (%SEC-6-IPACCESSLOGP)
7 debugging Debug output

A common memory aid: Every Awesome Cisco Engineer Will Need Ice-cream Daily. Setting a level sends that level and everything with a lower number.

Message format

*Oct  7 10:15:32.118: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2, changed state to down
 timestamp           facility-severity-mnemonic: description

Destinations

Destination Command Default
Console logging console <level> On, debugging (7)
VTY sessions logging monitor <level>, then terminal monitor in each session On, but each session must run terminal monitor
RAM buffer logging buffered [size] <level> Enabled on IOS XE; read with show logging
Syslog server logging host <ip> and logging trap <level> Trap level informational (6), UDP 514 
service timestamps log datetime msec localtime show-timezone
service timestamps debug datetime msec localtime show-timezone
service sequence-numbers
!
logging buffered 64000 informational
logging console warnings
logging monitor debugging
logging host 192.168.100.50
logging trap notifications
logging source-interface Loopback0
logging origin-id hostname

logging source-interface makes every message come from one stable address, so the server can identify the device even when the exit interface changes. Accurate timestamps need NTP; without it, log entries from different devices cannot be correlated.

Exam trap

logging trap sets the level sent to the syslog server. It has nothing to do with SNMP traps. And because each destination has its own level, a message can appear in show logging (buffer at informational) but never reach the server (trap at warnings).

SNMP

SNMP lets a Network Management System (NMS) read and write values in the device's MIB, and lets the device send unsolicited notifications.

Message Direction Port Acknowledged
Get, GetNext, GetBulk (v2c+) NMS to agent UDP 161 Response
Set NMS to agent UDP 161 Response
Trap Agent to NMS UDP 162 No
Inform (v2c+) Agent to NMS UDP 162 Yes, retransmitted until acknowledged

Versions and security levels

Version Authentication Encryption Notes
v1 Community string, cleartext None Obsolete
v2c Community string, cleartext None Adds GetBulk and informs
v3 noAuthNoPriv Username only None Keyword noauth
v3 authNoPriv HMAC (MD5 or SHA) None Keyword auth
v3 authPriv HMAC (SHA) DES, 3DES or AES Keyword priv; the one to use 
ip access-list standard NMS-ONLY
 permit 192.168.100.50
!
snmp-server community R3ad0nly RO NMS-ONLY
snmp-server location DC1-Rack4
snmp-server contact [email protected]
snmp-server host 192.168.100.50 version 2c R3ad0nly
snmp-server enable traps

snmp-server view ALL iso included
snmp-server group NETOPS v3 priv read ALL
snmp-server user monitor NETOPS v3 auth sha Auth#Pass1 priv aes 128 Priv#Pass1
snmp-server host 192.168.100.50 version 3 priv monitor
snmp-server enable traps

The SNMPv3 hierarchy is: a view limits which OIDs are visible, a group sets the security level and the read/write/notify views, and a user belongs to a group and carries the keys. Verify with:

R1# show snmp group
groupname: NETOPS                           security model:v3 priv
contextname: <no context specified>         storage-type: nonvolatile
readview : ALL                              writeview: <no writeview specified>
notifyview: <no notifyview specified>
row status: active

R1# show snmp user
User name: monitor
Engine ID: 800000090300500000030000
storage-type: nonvolatile        active
Authentication Protocol: SHA
Privacy Protocol: AES128
Group-name: NETOPS

Exam trap

SNMPv3 users do not appear in show running-config, because the keys are localized to the engine ID. Use show snmp user. A question that shows a running config with an snmp-server group but no snmp-server user line is not necessarily missing the user.

Exam trap

snmp-server host controls where notifications go. It does not permit or deny polling. Polling access is controlled by the community ACL (v2c) or the user, group and security level (v3).

Debugs and conditional debugs

debug output is generated by the router's CPU in process context. On a busy device, an unrestricted debug can cause high CPU and even lose you the session. Rules of thumb:

  1. Send debug output to the buffer, not the console: logging buffered 128000 debugging and no logging console.
  2. Use timestamps with milliseconds so events can be ordered.
  3. Restrict the debug with a condition or an ACL before you turn it on.
  4. Have undebug all (or u all) ready to type.

Conditional debug

debug condition limits conditional-aware debugs to matching interfaces, addresses, usernames and similar attributes.

R1# debug condition interface GigabitEthernet2
Condition 1 set
R1# debug condition ip 10.1.12.2
Condition 2 set
R1# debug ip ospf adj
R1# show debugging condition
Condition 1: interface Gi2 (1 flags triggered)
        Flags: Gi2
Condition 2: ip 10.1.12.2 (0 flags triggered)
R1# undebug all
R1# no debug condition all

Some debugs also take an ACL directly, which is the classic way to limit packet debugs:

R1(config)# access-list 150 permit icmp host 10.10.10.11 host 10.255.255.3
R1# debug ip packet 150 detail

debug ip packet only shows process-switched packets. Traffic that CEF switches in the data plane is not shown, which is why the platform packet tracer exists on IOS XE.

debug condition interface GigabitEthernet2
debug ip routing
show debugging

debug platform condition ipv4 10.10.10.11/32 both
debug platform packet-trace packet 128
debug platform condition start
! generate traffic
debug platform condition stop
show platform packet-trace summary
show platform packet-trace packet 0
clear platform condition all

The packet tracer follows packets through the data plane and reports the result per packet (for example FWD or DROP with a drop reason), so it sees what debug ip packet cannot.

Exam trap

When you remove the last interface condition, IOS warns that all debugging becomes unconditional and may flood the device. Turn the debug off with undebug all before removing the conditions, not after.

Lab: logging, SNMPv3 and a conditional debug

Topology: R1 and R2 (Catalyst 8000v) joined on Gi2, subnet 10.1.12.0/30, OSPF area 0 including both loopbacks. R1 Gi1 is in the management subnet 192.168.100.0/24 with an Ubuntu NMS at 192.168.100.50 running rsyslog (UDP 514 enabled) and the snmp package.

graph LR
  NMS["NMS 192.168.100.50"] -- "Gi1 192.168.100.1/24" --- R1["R1 Lo0 10.255.255.1"]
  R1 -- "Gi2 10.1.12.1/30" --- R2["R2 Lo0 10.255.255.2"]
  1. On R1, configure timestamps, the buffer and the syslog server from the Syslog section, with logging trap notifications.
  2. Configure the SNMPv3 example. From the NMS, run snmpwalk -v3 -l authPriv -u monitor -a SHA -A 'Auth#Pass1' -x AES -X 'Priv#Pass1' 192.168.100.1 1.3.6.1.2.1.1 and check you receive the system group.
  3. Run the same walk with -l authNoPriv. It must fail, which proves the group level is enforced.
  4. On R1, debug condition interface GigabitEthernet2, then debug ip ospf adj. On R2, shutdown and no shutdown Gi2. Read the output with show logging.
  5. undebug all, then no debug condition all.
  6. Ping R2's loopback from R1 with source Loopback0 size 1500 df-bit, then lower R2 Gi2 to ip mtu 1400 and repeat. Note the change from ! to M or ..

Expected verification:

R1# show logging | include Logging to|Trap
    Logging to 192.168.100.50  (udp port 514, audit disabled,
         link up),
    Trap logging: level notifications, 14 message lines logged

R1# show logging | include OSPF-5
*Oct  7 10:21:04.511 UTC: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.255.2 on GigabitEthernet2 from FULL to DOWN, Neighbor Down: Interface down or detached
*Oct  7 10:21:41.902 UTC: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.255.2 on GigabitEthernet2 from LOADING to FULL, Loading Done

On the NMS, tail -f /var/log/syslog shows the same %OSPF-5-ADJCHG lines arriving from 10.255.255.1, the loopback set by logging source-interface.

For a timed version of this lab with SPAN and ERSPAN added, do Sim 10.

Quick check

logging trap warnings is configured. Is %LINEPROTO-5-UPDOWN sent to the syslog server?

No. LINEPROTO is severity 5 (notifications). Warnings is level 4, so only levels 0 to 4 are sent.

An extended ping with the DF bit set returns M.M.M. What is the problem?

A link on the path has an MTU smaller than the packet size. The router there cannot fragment the packet because DF is set, so it returns ICMP "fragmentation needed", shown as M.

What is the difference between an SNMP trap and an inform?

Both are notifications sent by the agent to UDP 162. A trap is not acknowledged. An inform is acknowledged by the NMS and retransmitted until it is, so it is reliable but uses more resources. Informs need v2c or v3.

References