Skip to content
Log in Sign up My account

4.3 Troubleshooting with Logging and Debugs

Blueprint 4.3: Troubleshoot network problems using logging

  • Local logging: console, monitor (VTY), buffer
  • Syslog to a remote server
  • Debugs and conditional debugs
  • Timestamps

Two kinds of question appear. One shows a logging configuration and asks why a message does not reach a destination (almost always a severity level). The other shows a debug that prints nothing, or prints too much, and asks why.

Severity levels

Every IOS message has a severity from 0 to 7. A destination configured at level n receives that level and everything more severe (lower numbers).

Level Keyword Example message
0 emergencies System unusable
1 alerts Immediate action needed, for example a temperature alarm
2 critical Critical condition, for example a memory allocation failure
3 errors %LINK-3-UPDOWN
4 warnings Conditions worth attention that are not yet errors
5 notifications %LINEPROTO-5-UPDOWN, %OSPF-5-ADJCHG, %SYS-5-CONFIG_I, %BGP-5-ADJCHANGE
6 informational %SEC-6-IPACCESSLOGP (ACL log entries)
7 debugging Output of debug commands

A common memory aid: Every Awesome Cisco Engineer Will Need Ice cream Daily.

Reading a message

*Oct  7 10:15:22.123 UTC: %OSPF-5-ADJCHG: Process 1, Nbr 10.255.255.2 on GigabitEthernet1 from FULL to DOWN, Neighbor Down: Dead timer expired
| |                       |    | |
| timestamp               |    | mnemonic
clock flag                |    severity
                          facility

The character before the timestamp says how far to trust the clock:

Prefix Meaning
none Clock is synchronised (NTP) and authoritative
* Clock is not authoritative: never set, or set manually and not trusted
. Clock was synchronised but has lost synchronisation

Exam trap

The facility in a message (OSPF, LINK) is not the syslog facility sent to the server. IOS sends every message to the syslog server with facility local7 by default (changed with logging facility). Questions sometimes mix the two up.

Logging destinations

graph LR
  MSG["Log message<br/>severity 0 to 7"] --> CON["Console<br/>logging console LEVEL"]
  MSG --> MON["VTY sessions<br/>logging monitor LEVEL<br/>plus terminal monitor"]
  MSG --> BUF["RAM buffer<br/>logging buffered SIZE LEVEL"]
  MSG --> SYS["Syslog server UDP 514<br/>logging host IP<br/>logging trap LEVEL"]
  MSG --> SNMP["SNMP trap<br/>logging history LEVEL"]
Destination Command Default level Notes
Console logging console LEVEL debugging Slow (serial speed). Heavy debug output to the console can make a router unresponsive.
Monitor (VTY) logging monitor LEVEL debugging Each SSH session must also run terminal monitor, every time it connects
Buffer logging buffered SIZE LEVEL debugging Read with show logging; lost on reload; cleared with clear logging
Syslog server logging host IP plus logging trap LEVEL informational UDP 514 by default; logging source-interface sets the source 
service timestamps log datetime msec localtime show-timezone year
service timestamps debug datetime msec localtime show-timezone year
service sequence-numbers
clock timezone UTC 0 0
ntp server 10.255.255.2
!
logging buffered 64000 informational
logging console warnings
logging monitor informational
logging host 192.168.100.50
logging trap notifications
logging source-interface Loopback0

Reading show logging

R1# show logging
Syslog logging: enabled (0 messages dropped, 3 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

    Console logging: level warnings, 12 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level informational, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level informational, 214 messages logged, xml disabled,
                    filtering disabled
    Trap logging: level notifications, 198 message lines logged
        Logging to 192.168.100.50  (udp port 514, audit disabled,
              link up),
              198 message lines logged,
              0 message lines rate-limited,
              0 message lines dropped-by-MD,
              xml disabled, sequence number disabled
              filtering disabled
        Logging Source-Interface:       VRF Name:
        Loopback0

Log Buffer (64000 bytes):

The header answers most exam questions: the level per destination, whether the syslog host is defined, its port, and which source interface is used. Monitor logging: ... 0 messages logged with a level set usually means nobody ran terminal monitor.

Exam trap

logging trap warnings sends levels 0 to 4. OSPF and BGP neighbour changes and interface line-protocol changes are level 5 (notifications), so they never reach the server, while %LINK-3-UPDOWN (level 3) does. When a question says "some messages arrive and others do not", compare the missing message's severity number with the trap level.

Exam trap

Timestamps in uptime format (00:12:34: or 2d04h:) or with a leading * cannot be correlated across devices. The complete fix is service timestamps log datetime msec and a synchronised clock from NTP. Changing the timestamp format alone still leaves every router with its own idea of the time.

Debugs

Debugs print level 7 messages for one protocol or feature. They are powerful, CPU-hungry and the main way to see why something fails (authentication mismatch, wrong hello timer, DHCP pool not found).

Running a debug safely

  1. Send debugs to the buffer, not the console: no logging console (or logging console warnings) and logging buffered 64000 debugging.
  2. Turn on service timestamps debug datetime msec so you can sequence events.
  3. Narrow the debug as far as it goes: an ACL, a neighbour, an interface, or a debug condition.
  4. Have the off switch ready: undebug all (or u all). On a slow console session, type it before you start the debug in a second session.
  5. Confirm what is running with show debugging.
R1# configure terminal
R1(config)# no logging console
R1(config)# logging buffered 64000 debugging
R1(config)# end
R1# debug ip ospf adj
R1# show logging | include OSPF
R1# undebug all

Why debug ip packet shows nothing

debug ip packet only sees packets that are process switched, which means packets sent to or from the router itself (pings from the router, routing protocol traffic, management sessions). Transit traffic is CEF switched in the data plane and never reaches the debug. A user pinging through R2 produces no output on R2.

access-list 150 permit icmp host 10.10.10.10 host 192.168.100.50
access-list 150 permit icmp host 192.168.100.50 host 10.10.10.10
R2# debug ip packet 150 detail

To see transit traffic, use a capture tool instead:

R2# monitor capture CAP interface GigabitEthernet0/0 both
R2# monitor capture CAP match ipv4 host 10.10.10.10 any
R2# monitor capture CAP start
R2# show monitor capture CAP buffer brief
R2# monitor capture CAP stop
Supported on IOS XE (Catalyst 8000v). Captures real transit packets without process switching them.

R1# debug platform condition ipv4 10.10.10.10/32 both
R1# debug platform packet-trace packet 16
R1# debug platform condition start
R1# show platform packet-trace summary
R1# debug platform condition stop
R1# clear platform condition all
Shows each matched packet's input and output interface and its disposition (forwarded or dropped, with a drop reason). Available on routing platforms such as Catalyst 8000v, ISR 4000 and ASR 1000.

Exam trap

"Pings through the router succeed but debug ip packet prints nothing" is not a fault. The traffic is CEF switched. Answers that suggest no ip route-cache to force process switching belong to old IOS practice and are risky in production; on IOS XE the answer is EPC or packet trace.

Conditional debugging

Conditional debugging limits the output of supported debugs to one interface, one user, one IP address and similar. Once you set the first condition, output from supported debugs is printed only when a condition matches.

R1# debug condition interface GigabitEthernet1
R1# debug condition username alice
R1# show debugging
R1# undebug condition 1          ! remove one condition by number
R1# undebug condition all        ! remove all conditions: unfiltered output follows

Points to remember:

  • Not every debug honours conditions. Those tied to interfaces, sessions or users (PPP, AAA, many tunnel and session debugs) do; others print regardless. Always check with a quick test before trusting the filter.
  • Removing the last condition switches the filter off. On a busy router with debugs still running, undebug condition all turns a trickle into a flood. IOS warns and asks for confirmation. Run undebug all first.
  • debug ip packet ACL and protocol options such as debug ip bgp 10.1.2.2 updates are filters built into the debug itself, not conditions.

A logging-based troubleshooting flow

graph TB
  S["Symptom reported"] --> L["show logging | include keyword<br/>what happened, when"]
  L --> T["Timestamps trustworthy?<br/>show clock detail, show ntp status"]
  T --> C["Correlate with neighbours<br/>syslog server, same time window"]
  C --> D["Targeted debug<br/>to buffer, filtered"]
  D --> F["Fix, then confirm<br/>the log shows the recovery"]

show clock detail shows the time source (Time source is NTP or No time source). show ntp status should show Clock is synchronized. If the clock is not synchronised, fix NTP before trusting any timestamps.

Lab: logging and debug break/fix

Topology: module lab. R1, R2 and the NMS running rsyslog listening on UDP 514 (module(load="imudp") and input(type="imudp" port="514") in /etc/rsyslog.conf).

  1. Configure the logging block above on R1 and R2, with R2 as the NTP server (ntp master 3 on R2 is fine in the lab).
  2. Verify with show ntp status (synchronised), show logging (trap level notifications, host 192.168.100.50) and tail -f /var/log/syslog on the NMS while you flap R1 Gi3.
  3. Save the baseline.
  4. Fault A. Set logging trap warnings on R1. Clear the OSPF process with clear ip ospf process. The syslog server receives no OSPF messages, although the local buffer shows them. Explain using the severity table, then restore notifications.
  5. Fault B. Remove ntp server on R1 and set the clock wrong with clock set. Note the * before new log timestamps. Restore NTP and wait for . to disappear and synchronisation to return.
  6. Fault C. Create Loopback99 (172.31.99.1/32, not advertised) and set logging source-interface Loopback99. Messages still reach the NMS, because UDP syslog needs no return path, but they now arrive from 172.31.99.1. Add an rsyslog rule on the NMS that only accepts 10.255.255.1 and watch R1's messages disappear. This is the real-world version of the fault: the server filters on the expected source. Restore Loopback0.
  7. Debug practice. From HOST1 ping the NMS (HOST1 can use a static address 10.10.10.10/24 until the DHCP lab). On R1 run debug ip packet and confirm there is no output for the transit pings. Then run the Embedded Packet Capture commands on R1 Gi2 (R1 is the Catalyst 8000v) and see the packets.
  8. Conditional debug. On R1 run debug condition interface GigabitEthernet1 and debug ip ospf adj, then clear OSPF. Compare the output to the same debug without the condition and record whether this debug honours the condition on your image.

Expected verification output:

R1# show ntp status
Clock is synchronized, stratum 4, reference is 10.255.255.2
...
R1# show logging | include Trap|Logging to
    Trap logging: level notifications, 41 message lines logged
        Logging to 192.168.100.50  (udp port 514, audit disabled,

Quick check

logging trap informational is set. Which of these reach the syslog server: %SYS-5-CONFIG_I, %SEC-6-IPACCESSLOGP, debug output?

The first two (levels 5 and 6). Debug output is level 7, which is above informational, so it stays local unless the trap level is debugging.

An engineer connects with SSH, runs debug ip ospf hello and sees nothing, though show logging shows the debug messages in the buffer. What did they forget?

terminal monitor. VTY sessions do not display log or debug messages unless the session runs it, even when logging monitor is set.

Log messages from R5 start with .Oct 7 03:12:44.512. What does the leading dot mean, and what do you check?

The clock was synchronised but has lost synchronisation. Check show ntp associations and show ntp status for reachability to the NTP server and any authentication or access-group problem.

References

  • Cisco IOS XE System Message Logging Configuration Guide: "Configuring System Message Logging" (search on cisco.com)
  • Cisco IOS Debug Command Reference and "Conditionally Triggered Debugging" (search on cisco.com)
  • Cisco IOS XE "Embedded Packet Capture" and "Packet Trace" configuration guides (search on cisco.com)
  • RFC 5424 (The Syslog Protocol): rfc-editor.org/rfc/rfc5424
  • Cisco Live On-Demand Library: search "IOS XE troubleshooting packet trace"