Skip to content
Log in Sign up My account

4.1 Troubleshooting Device Management

Blueprint 4.1: Troubleshoot device management

  • 4.1.a Console and VTY
  • 4.1.b Telnet, HTTP, HTTPS, SSH, SCP
  • 4.1.c (T)FTP

The verb is troubleshoot. Expect a config fragment and an error message (Password required, but none set, % Connection refused by remote host, SSH Disabled) and a question about which line is missing or wrong.

The access chain

Every management session passes through the same gates in the same order. When access fails, walk the chain and stop at the first gate that does not pass.

graph TB
  A["Packet reaches router<br/>route back to client?"] --> B["Interface ACL or CoPP<br/>permits TCP 22 or 23?"]
  B --> C["Line access-class<br/>permits source? vrf-also?"]
  C --> D["transport input<br/>allows this protocol?"]
  D --> E["Service ready?<br/>SSH needs RSA key"]
  E --> F["Login method<br/>login, login local, AAA list"]
  F --> G["Privilege<br/>enable secret or privilege 15"]
  G --> H["Session up"]
Gate Command that controls it Command that proves it
Reachability Routing, source interface of the client ping, traceroute from the router to the client
Interface filter ip access-group on the ingress interface, CoPP show ip access-lists (hit counters), show policy-map control-plane
Line filter access-class under line vty show running-config \| section line vty, show ip access-lists
Protocol transport input show line vty 0 (lists allowed input transports)
Service crypto key generate rsa, ip http server, ip scp server enable show ip ssh, show ip http server status
Authentication login, login local, login authentication LIST show users, debug aaa authentication
Authorisation enable secret, username ... privilege 15, privilege level 15 show privilege after login

Console and VTY lines

IOS XE has one console line (line con 0), on some platforms an auxiliary line, and a set of virtual terminal lines for remote sessions. Most platforms have 16 VTY lines, usually shown in the config as two blocks: line vty 0 4 and line vty 5 15. Each block is configured separately. A security setting on 0 to 4 does nothing for 5 to 15.

line con 0
 exec-timeout 15 0            ! default is 10 minutes; 0 0 disables the timeout
 logging synchronous          ! log messages do not break the line you are typing
 login local
line vty 0 4
 access-class MGMT-ONLY in
 exec-timeout 10 0
 login local
 transport input ssh
line vty 5 15
 access-class MGMT-ONLY in
 exec-timeout 10 0
 login local
 transport input ssh
!
ip access-list standard MGMT-ONLY
 permit 192.168.100.0 0.0.0.255

Login methods

Line command What it checks Failure symptom
no login Nothing. Anyone reaching the line gets in. No failure, which is the problem
login with password X The line password login with no password on a VTY line: Telnet shows Password required, but none set and closes
login local The local username database No matching username or wrong secret: % Login invalid
login authentication LIST An AAA method list (needs aaa new-model) List not defined or server down with no fallback: login fails. See 3.0 Infrastructure Security

Once aaa new-model is enabled, the line password and login commands are no longer what decides authentication. AAA method lists take over. A config that shows both is a frequent exhibit: the line password is ignored, and the question is really about the method list.

Getting to privileged EXEC

A remote user lands in user EXEC (privilege 1) unless something raises the level. Typing enable over a VTY session on a router with no enable secret or enable password returns % No password set. The console is different: with no enable password, enable on the console just works. That is why a router that "works on the console but not over SSH" often has no enable secret.

Three ways to reach privilege 15 over VTY:

  • enable secret and the user types enable
  • username admin privilege 15 secret ... with login local, which drops the user straight into privileged EXEC
  • privilege level 15 under the line, which applies to everyone using that line

access-class

access-class ACL in under a line filters which source addresses may open a session to that line. It takes a standard or extended ACL and it is applied under the line, not on an interface. access-class ACL out is different: it filters which destinations a user already logged in may Telnet or SSH to from the router.

If the management interface is in a VRF (for example GigabitEthernet0 in Mgmt-intf on ISR and ASR routers, or Mgmt-vrf on Catalyst 9000 switches), a plain access-class only matches sessions arriving in the global table. Sessions from the VRF are refused. Add the vrf-also keyword:

line vty 0 15
 access-class MGMT-ONLY in vrf-also

Exam trap

An exhibit shows access-class on line vty 0 4 only. Five administrators log in normally, and the sixth session comes from an unauthorised address and succeeds. The sixth session lands on VTY 5, which has no access-class. Apply the same filter to every VTY block, or configure line vty 0 15 in one go.

Exam trap

access-class filters by source for inbound sessions. A standard ACL that permits the router's own management address instead of the administrators' subnet blocks everyone. Read the ACL against the client's address, not the router's.

Remote access protocols

Protocol Transport Secure Server-side requirement
Telnet TCP 23 No, clear text transport input telnet (or all) and a login method
SSH TCP 22 Yes Hostname, domain name, RSA key, transport input ssh
HTTP TCP 80 No ip http server
HTTPS TCP 443 Yes ip http secure-server (uses a self-signed or configured trustpoint certificate)
SCP TCP 22 (over SSH) Yes Working SSH, ip scp server enable, AAA authentication and exec authorisation
TFTP UDP 69 No, no authentication TFTP server reachable; on IOS, tftp-server flash:file to serve a file
FTP TCP 21 control, TCP 20 or high port for data No Credentials with ip ftp username/ip ftp password or in the URL

transport input

transport input on the VTY lines decides which protocols may open a session. Defaults differ between platforms and releases, so never assume: check the running configuration or show line vty 0. When a protocol is not allowed, the client sees % Connection refused by remote host for Telnet or a refused or reset connection for SSH.

transport input ssh telnet allows both. transport input none blocks all remote sessions while still letting the router open outbound sessions. transport output is a different command that controls which protocols a logged-in user may use from the router.

SSH

SSH needs an RSA key pair. The default key name is built from the hostname and the domain name, so both must exist before the key can be generated:

hostname R1
ip domain name lab.example
crypto key generate rsa modulus 2048
ip ssh version 2
ip ssh source-interface Loopback0        ! optional: source for outbound SSH from R1
username admin privilege 15 secret Cisc0Lab!
line vty 0 15
 login local
 transport input ssh

crypto key generate rsa with the default hostname Router or without a domain name fails with an error asking for a host name and domain name. SSH version 2 needs a key of at least 768 bits; use 2048. Removing the key with crypto key zeroize rsa disables SSH immediately.

R1# show ip ssh
SSH Enabled - version 2.0
Authentication methods:publickey,keyboard-interactive,password
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 2048 bits

R1# show ssh
Connection Version Mode Encryption  Hmac         State           Username
0          2.0     IN   aes256-ctr  hmac-sha2-256 Session started admin
0          2.0     OUT  aes256-ctr  hmac-sha2-256 Session started admin

The broken version reads like this:

R1# show ip ssh
SSH Disabled - version 1.99
%Please create RSA keys to enable SSH (and of atleast 768 bits for SSH v2).

version 1.99 means "1 and 2 both allowed" and is what you see when ip ssh version is not set. It is not a fault on its own, but the second line is. Useful verification and debug commands: show crypto key mypubkey rsa, show users, debug ip ssh.

Exam trap

ip ssh version 2 on its own does not enable SSH. If show ip ssh says Disabled, the RSA key is missing, and the answer that adds a domain name and generates the key is the right one. Watch for answers that add transport input ssh when the lines already allow SSH.

HTTP and HTTPS

The web server is used by WebUI on IOS XE and by RESTCONF. It is off unless enabled.

ip http secure-server
no ip http server                     ! disable clear-text HTTP
ip http authentication local          ! use the local username database
ip http access-class 10               ! standard ACL of allowed clients
access-list 10 permit 192.168.100.0 0.0.0.255

Newer IOS XE releases also accept named ACLs with ip http access-class ipv4 NAME. Prove it with show ip http server status, which lists whether the HTTP and HTTPS servers are enabled, their ports, the authentication method and the access class. Failure modes:

  • HTTPS server enabled but the browser cannot connect: check the ACL in ip http access-class and any interface ACL for TCP 443.
  • Login prompt rejects valid users: ip http authentication points at a method (enable password, AAA list) the user is not in.
  • RESTCONF requests fail with the web server up: RESTCONF itself is not enabled (restconf), which is an ENCOR 4.6 topic.

SCP

SCP runs inside SSH, so SSH must work first. The IOS SCP server then needs AAA so it can authorise the user to run commands:

aaa new-model
aaa authentication login default local
aaa authorization exec default local
username admin privilege 15 secret Cisc0Lab!
ip scp server enable

From a Linux host: scp ios.bin [email protected]:flash:/ios.bin. From the router as a client: copy scp://[email protected]/backup.cfg flash:.

Exam trap

SSH logins work but SCP transfers fail. The usual missing pieces are ip scp server enable and aaa authorization exec. The user also needs privilege 15. SSH login alone only proves authentication, not authorisation.

TFTP and FTP

The router is usually the client, copying images or configurations to and from a server.

ip tftp source-interface Loopback0
ip ftp source-interface Loopback0
ip ftp username backup
ip ftp password Bkp0Pass
!
R1# copy running-config tftp://192.168.100.50/R1.cfg
R1# copy ftp://192.168.100.50/c8000v.bin bootflash:
R1# verify /md5 bootflash:c8000v.bin

UDP 69, no authentication. Many TFTP servers only accept writes to files that already exist, or only from permitted addresses. IOS can also act as a server for files in flash with tftp-server flash:FILE and an optional ACL.

TCP 21 for control plus a data connection. Needs credentials. Without ip ftp username, IOS tries an anonymous login. Credentials can also go in the URL: ftp://user:pass@host/file.

Encrypted and authenticated, both over SSH. Use them for anything sensitive, such as configuration backups that contain secrets.

Common failure messages and what they mean:

Symptom Likely cause
%Error opening tftp://... (Timed out) No route to the server, UDP 69 filtered, server not running, or the server's ACL rejects the router's source address
(No such file or directory) Wrong path or file name, or the server refuses to create new files
Transfer works from one router but not another The router sources the request from a different interface than the server allows; set ip tftp source-interface
Image copied but fails to boot Corrupt or incomplete transfer; check with verify /md5 against the published hash
Copy to flash fails part way Not enough space; check with dir bootflash: or show file systems

Exam trap

Without ip tftp source-interface (or ip ftp source-interface), the router uses the address of the exit interface toward the server. If the server only permits the router's loopback, transfers time out even though ping to the server works.

Lab: break/fix management access

Topology: module lab, R1 and the NMS. OSPF gives the NMS (192.168.100.50) a path to R1 Lo0.

  1. Build the working baseline on R1:

    hostname R1
ip domain name lab.example
crypto key generate rsa modulus 2048
ip ssh version 2
aaa new-model
aaa authentication login default local
aaa authorization exec default local
username admin privilege 15 secret Cisc0Lab!
ip scp server enable
ip access-list standard MGMT-ONLY
 permit 192.168.100.0 0.0.0.255
line vty 0 4
 access-class MGMT-ONLY in
 transport input ssh
line vty 5 15
 access-class MGMT-ONLY in
 transport input ssh

  2. From the NMS run ssh [email protected] and scp /etc/hostname [email protected]:flash:/test.txt. Both should work. On R1 run show ssh and show users.

  3. Save the baseline: copy running-config flash:baseline.cfg.
  4. Fault A. Run crypto key zeroize rsa. Retry SSH from the NMS. Check show ip ssh. Fix by regenerating the key.
  5. Fault B. Under line vty 0 15 set transport input telnet. SSH from the NMS is now refused while Telnet reaches a login prompt. Run show line vty 0 and find the allowed input transports. Restore transport input ssh.
  6. Fault C. Change the ACL to permit host 10.255.255.1. SSH from the NMS is refused. Run show ip access-lists MGMT-ONLY and note the zero match counter on the permit line. Fix it.
  7. Fault D. Remove aaa authorization exec default local. SSH logins still work (for a privilege 15 user), but SCP fails. Restore it.
  8. On R1 run copy running-config tftp://192.168.100.50/R1.cfg with a TFTP server running on the NMS (tftpd-hpa). Then add ip tftp source-interface Loopback0 and confirm with tcpdump -n udp port 69 on the NMS that the source changes to 10.255.255.1.

Expected verification output (baseline):

R1# show ip access-lists MGMT-ONLY
Standard IP access list MGMT-ONLY
    10 permit 192.168.100.0, wildcard bits 0.0.0.255 (4 matches)

R1# show users
    Line       User       Host(s)              Idle       Location
*  1 vty 0     admin      idle                 00:00:00 192.168.100.50

Quick check

SSH to R4 is refused. show ip ssh shows SSH Enabled - version 2.0, and the VTY lines have transport input ssh and login local. What else would you check, in order?

The access-class on every VTY block and the ACL it references (is the client's source permitted, and is vrf-also needed if the client is in a VRF), then any interface ACL or CoPP policy that might drop TCP 22, and finally reachability back to the client.

An administrator logs in with SSH and types enable. The router answers % No password set. What is wrong?

No enable secret (or enable password) is configured. On a VTY line IOS refuses to enter privileged EXEC without one. Configure an enable secret, or give the user privilege 15 in the local database.

Why might copy running-config tftp: time out on R1 while ping 192.168.100.50 succeeds?

The TFTP server may only accept the router's loopback (the ping and the TFTP request use the exit interface address by default), UDP 69 may be filtered on the path, or the TFTP service is not running or refuses to create new files. Set ip tftp source-interface and check the server.

References

  • Cisco IOS XE Security Configuration Guide: Secure Shell (search "Configuring Secure Shell" on cisco.com)
  • Cisco IOS XE Configuration Fundamentals Configuration Guide: "Managing Connections, Menus and System Banners" and "Using the Cisco IOS File System" (search on cisco.com)
  • Cisco IOS XE HTTP Services Configuration Guide: "HTTP 1.1 Web Server and Client" (search on cisco.com)
  • RFC 4253 (SSH transport layer protocol): rfc-editor.org/rfc/rfc4253
  • RFC 1350 (TFTP): rfc-editor.org/rfc/rfc1350
  • Cisco Live On-Demand Library: search "management plane security IOS XE"