Skip to content
Log in Sign up My account

4.5 Troubleshooting with IP SLA and Object Tracking

Blueprint 4.5: Troubleshoot network performance issues using IP SLA

  • Jitter
  • Tracking objects
  • Delay
  • Connectivity

Expect exhibits of show ip sla statistics, show ip sla configuration and show track, usually next to a static route that is not behaving. The common faults are an operation that never started, a probe that follows the very route it is supposed to test, and a jitter operation with no responder.

What IP SLA does

IP SLA makes the router generate synthetic test traffic, measure the result and keep statistics. Because the traffic is generated on a schedule, it measures the network continuously even when users are idle, and it can feed object tracking, which in turn controls static routes, HSRP priority, PBR next hops and EEM applets.

Operation Measures Responder needed? Blueprint word
icmp-echo Reachability and round-trip time No Connectivity, delay
icmp-jitter RTT, jitter and loss using ICMP timestamps No Jitter
udp-echo RTT with UDP Only for ports with no real service Delay
udp-jitter RTT, one-way delay, jitter each way, loss each way, MOS for voice codecs Yes, a Cisco IP SLA responder Jitter, delay
tcp-connect Time to open a TCP connection Only for ports with no real service Connectivity
http, dns, dhcp Application response times No (real servers) Connectivity
path-echo, path-jitter Hop-by-hop RTT No Delay

Configuring and starting an operation

ip sla 10
 icmp-echo 10.1.2.2 source-interface GigabitEthernet1
 frequency 10                 ! seconds, default 60
 timeout 2000                 ! milliseconds, default 5000
 threshold 500                ! milliseconds, default 5000
ip sla schedule 10 life forever start-time now

Three defaults to remember: frequency 60 seconds, timeout 5000 ms, threshold 5000 ms. The timeout must not be longer than the frequency, and the threshold should not be longer than the timeout, or IOS rejects or warns about the setting.

Exam trap

An ip sla definition does nothing until it is scheduled. With no ip sla schedule line, show ip sla statistics shows no successes and no failures, the tracked object stays down, and a static route tied to it never appears. Look for the schedule line first in any IP SLA exhibit.

A configured operation cannot be edited while it is scheduled on many releases. To change it, remove it with no ip sla 10, then re-enter the definition and the schedule.

Reading the statistics

R1# show ip sla statistics 10
IPSLAs Latest Operation Statistics

IPSLA operation id: 10
        Latest RTT: 2 milliseconds
Latest operation start time: 09:12:40 UTC Thu Oct 8 2026
Latest operation return code: OK
Number of successes: 361
Number of failures: 4
Operation time to live: Forever
Return code Meaning
OK Probe succeeded within the threshold
Over threshold Probe succeeded, but the RTT was above threshold
Timeout No reply within timeout
No connection Could not open the connection (TCP and responder-based operations)

show ip sla configuration 10 shows the full definition including defaults (frequency, timeout, threshold, schedule state). show ip sla summary gives one line per operation with its last return code. debug ip sla trace and debug ip sla error show each probe.

udp-jitter and the responder

udp-jitter sends a train of UDP packets (10 packets 20 ms apart by default) to a responder, which timestamps and returns them. It reports round-trip time, jitter in each direction, packet loss in each direction, out-of-sequence and late packets, and, with a codec configured, ICPIF and MOS voice quality scores.

! R2, the target
ip sla responder
!
! R1, the source
ip sla 20
 udp-jitter 10.255.255.2 16384 source-ip 10.255.255.1 codec g711ulaw
 tos 184                      ! mark the probes EF so they follow the voice path through QoS
 frequency 30
ip sla schedule 20 life forever start-time now

The source first contacts the responder with a control message on UDP 1967 to tell it which port to open, then sends the test packets to that port. An ACL that permits the test port but not UDP 1967, or a target with no ip sla responder, makes the operation fail while an icmp-echo to the same address works.

R1# show ip sla statistics 20
IPSLA operation id: 20
Type of operation: udp-jitter
        Latest RTT: 4 milliseconds
Latest operation return code: OK
RTT Values:
        Number Of RTT: 1000             RTT Min/Avg/Max: 2/4/19 milliseconds
Latency one-way time:
        Number of Latency one-way Samples: 0
        Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
        Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
        Number of SD Jitter Samples: 999
        Number of DS Jitter Samples: 999
        Source to Destination Jitter Min/Avg/Max: 0/1/12 milliseconds
        Destination to Source Jitter Min/Avg/Max: 0/1/3 milliseconds
Packet Loss Values:
        Loss Source to Destination: 0
        Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0
        Packet Late Arrival: 0  Packet Skipped: 0
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 1
        Mean Opinion Score (MOS): 4.34
Number of successes: 12
Number of failures: 0

(Output trimmed; the exact layout varies by release. With a codec configured, the default packet train is larger than the plain 10-packet default, which is why there are 1000 RTT samples.)

How to read it: round-trip and jitter are fine; source-to-destination jitter peaks at 12 ms, which points to occasional congestion in the R1 to R2 direction. One-way latency samples are zero, because one-way delay needs the source and responder clocks synchronised (NTP). RTT and jitter do not need clock synchronisation, because each is calculated from timestamps taken on the same device or from differences between packets.

Exam trap

A question shows a working udp-jitter operation with zero one-way latency samples and asks why. The answer is clock synchronisation between source and responder, not a missing responder (if the responder were missing, there would be no RTT or jitter values at all).

Object tracking

A track object turns something measurable into a simple Up or Down state that other features can use.

Track type Up when
track N ip sla X reachability Return code is OK or Over threshold
track N ip sla X state Return code is OK only (over threshold counts as down)
track N interface Gi1 line-protocol Interface line protocol is up
track N ip route 10.0.0.0/8 reachability The route is in the RIB
track N list boolean and or or A combination of other track objects 
track 1 ip sla 10 reachability
 delay down 15 up 60          ! damp flaps: wait before declaring a change
!
ip route 0.0.0.0 0.0.0.0 10.1.2.2 track 1
ip route 0.0.0.0 0.0.0.0 10.1.3.2 250        ! floating static backup

graph LR
  SLA["ip sla 10<br/>icmp-echo 10.1.2.2"] --> TR["track 1<br/>reachability"]
  TR -->|"Up"| P["Default via 10.1.2.2<br/>AD 1 installed"]
  TR -->|"Down"| B["Default via 10.1.3.2<br/>AD 250 takes over"]
  TR --> H["standby 1 track 1 decrement 20"]
  TR --> E["EEM event track 1 state down"]
R1# show track 1
Track 1
  IP SLA 10 reachability
  Reachability is Up
    3 changes, last change 00:41:12
  Delay up 60 secs, down 15 secs
  Latest operation return code: OK
  Latest RTT (millisecs) 2
Tracked by:
  Static IP Routing 0

Tracked by confirms which clients use the object. A track object nobody references does nothing; a route that references a track number that does not exist is not installed.

Exam trap

reachability versus state: if the requirement is "fail over when latency exceeds 500 ms", tracking reachability never fails over, because Over threshold still counts as up. Use state with an appropriate threshold.

The recursive probe problem

The classic failure: R1 probes a target beyond the ISP (for example 8.8.8.8, or a head-office address) to detect upstream failures. The probe's own packets follow the routing table. When the primary default route is withdrawn, the probe now leaves through the backup path, succeeds, the track comes back up, the primary returns, the probe fails again, and the default route flaps.

graph TB
  A["Primary default via ISP-A fails upstream"] --> B["Probe times out, track 1 down"]
  B --> C["Primary route removed, backup installed"]
  C --> D["Probe now routed via ISP-B and succeeds"]
  D --> E["Track 1 up, primary reinstalled"]
  E --> A

source-interface does not fix this. It sets the source address, but the outgoing interface still comes from the routing table. The fixes:

  • A host route for the probe target through the primary next hop: ip route 192.0.2.10 255.255.255.255 10.1.2.2. If the primary next hop is down, the host route disappears and the probe fails as it should.
  • Or put the probe target on the primary path only (the provider's next hop), accepting that you then only test the first hop.

delay down and delay up on the track object only slow the flapping. They do not cure it.

Exam trap

When the tracked route and the probe target are behind the same default route, think recursion first. The answer is usually a static host route for the target via the primary next hop.

Troubleshooting checklist

Symptom Check Fix
Tracked route never installed show ip sla configuration: scheduled? show track: up? Add ip sla schedule, fix the probe target or source
Track always up during a real outage Probe target is the directly connected next hop, or reachability used for a latency requirement Probe a target beyond the failure point, use state
Route flaps every few probe intervals Probe follows the backup route when primary is withdrawn Host route for the probe target via the primary next hop
udp-jitter fails, icmp-echo works ip sla responder on target, ACL for UDP 1967 and test port Enable responder, permit the control and test ports
Jitter OK, one-way latency zero NTP on both ends Synchronise clocks
Probe ignores QoS path No tos set tos 184 (EF) for voice-like probes

Lab: IP SLA jitter with tracking

Topology: module lab. R1 primary path via Gi1 to R2, backup via Gi3 to R3. R2 Lo0 (10.255.255.2) stands in for an upstream target. For this lab, R1 uses static routes only, so the probe target is reached through the default route like an internet target would be.

  1. On R1 run no router ospf 1. On R2 add ip route 10.255.255.1 255.255.255.255 10.1.2.1 and ip route 10.255.255.1 255.255.255.255 10.2.3.2 250; on R3 add ip route 10.255.255.1 255.255.255.255 10.1.3.1. R2 and R3 keep OSPF between themselves, so R3 can reach R2 Lo0. On R2 configure ip sla responder.

  2. On R1 configure:

    ip sla 10
 icmp-echo 10.255.255.2 source-interface Loopback0
 frequency 5
 timeout 2000
ip sla schedule 10 life forever start-time now
track 1 ip sla 10 reachability
 delay down 10 up 20
ip route 0.0.0.0 0.0.0.0 10.1.2.2 track 1
ip route 0.0.0.0 0.0.0.0 10.1.3.2 250
ip sla 20
 udp-jitter 10.255.255.2 16384 source-ip 10.255.255.1 codec g711ulaw
 tos 184
 frequency 30
ip sla schedule 20 life forever start-time now

  3. Verify: show ip sla statistics, show track 1, show ip route 0.0.0.0. The primary default should be installed with AD 1.

  4. Save the baseline.
  5. Fault A. no ip sla schedule 10 (or remove and re-add the operation without a schedule). Watch the track go down and the backup route take over although the primary path is fine.
  6. Fault B. Shut R2 Gi0/0 (the far end of R1's primary link). R1 Gi1 stays up, so only the probe can detect the failure. Watch show track 1 and show logging: the track goes down, the backup default is installed, the probe now reaches 10.255.255.2 through R3, the track comes back up and the cycle repeats. Fix it with ip route 10.255.255.2 255.255.255.255 10.1.2.2 on R1 and confirm the track now stays down until you re-enable R2 Gi0/0.
  7. Fault C. Remove ip sla responder on R2. Operation 20 fails while operation 10 keeps working. Restore it.
  8. Fault D. Remove NTP from R1 (if you configured it in the logging lab) and check the one-way latency samples in operation 20. Then synchronise R1 and R2 and check again.

Expected verification output (baseline):

R1# show ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "static", distance 1, metric 0, candidate default path
  Routing Descriptor Blocks:
  * 10.1.2.2
      Route metric is 0, traffic share count is 1

Quick check

A track object references ip sla 5, and the SLA operation is configured and scheduled. show track says Reachability is Down and show ip sla statistics 5 shows Latest operation return code: Over threshold. Which keyword is the track using?

state. With reachability, Over threshold counts as up. Only state treats it as down.

Which IP SLA operation would you choose to measure one-way delay and jitter for voice between two branches, and what must be in place?

udp-jitter with a codec option, an IP SLA responder on the far router, UDP 1967 and the test port permitted along the path, and NTP on both routers for one-way delay values.

Why does adding source-interface GigabitEthernet1 to an icmp-echo probe not stop it from using the backup path?

The source interface sets the source address of the probe. The router still chooses the outgoing interface from the routing table, so when the primary route is removed the probe follows the backup default route. A host route for the target via the primary next hop pins it.

References

  • Cisco IOS XE IP SLAs Configuration Guide: "Configuring IP SLAs UDP Jitter Operations", "Configuring IP SLAs ICMP Echo Operations" and "Configuring Enhanced Object Tracking" (search on cisco.com)
  • Cisco "Reliable Static Routing Backup Using Object Tracking" (search on cisco.com)
  • RFC 3550 (RTP, interarrival jitter definition): rfc-editor.org/rfc/rfc3550
  • Cisco Live On-Demand Library: search "IP SLA"