Skip to content
Log in Sign up My account

3.1 Troubleshooting IOS AAA

Blueprint 3.1: Troubleshoot device security using IOS AAA (TACACS+, RADIUS, local database)

  • AAA building blocks: aaa new-model, server definitions, server groups, method lists
  • TACACS+ versus RADIUS for device administration
  • The local user database as a fallback, and how fallback really works (ERROR versus FAIL)
  • Authentication, authorization (EXEC and command) and accounting method lists
  • Fault symptoms and the show, test and debug commands that isolate them

The verb is troubleshoot. Expect exhibits with a method list and a symptom such as "the user lands at R1>", "local login no longer works" or "logins take five seconds and then fall back". You must read the configuration and say which line is wrong.

How IOS AAA is put together

AAA has four layers. Most faults are a missing or mismatched link between two of them.

Layer What it does Example command
Global switch Turns on the AAA engine and disables the old line-based model aaa new-model
Server Defines one AAA server: address, key, ports, timeout tacacs server TAC1, radius server RAD1
Server group Groups servers, sets source interface and VRF for the group aaa group server tacacs+ TAC-GRP
Method list Says which sources to try, in order, for one AAA function aaa authentication login VTY group TAC-GRP local
Attachment Ties a named list to a line or service login authentication VTY under line vty 0 15

A list named default needs no attachment. It applies to every line and service that does not reference a named list. A named list does nothing until something references it.

graph LR
  U["Admin SSH session"] --> L["line vty 0 15<br/>login authentication VTY"]
  L --> M["Method list VTY<br/>group TAC-GRP local"]
  M --> G["Server group TAC-GRP<br/>source Lo0"]
  G --> S1["tacacs server TAC1<br/>10.50.1.10 TCP 49"]
  G --> S2["tacacs server TAC2<br/>10.50.1.11 TCP 49"]
  M -.->|"only on ERROR"| LOC["Local username database"]

What aaa new-model changes

As soon as you enter aaa new-model, IOS stops using the line password and login commands for lines that AAA controls. If no default login list exists yet, VTY logins are checked against the local username database. On a router that has a VTY password but no username entries, that means remote access stops working the moment AAA is enabled. Build the local user first, then enable AAA.

Exam trap

login local and password under a line are ignored once aaa new-model is on. If an exhibit shows login local on the VTY lines and a default method list of group tacacs+, the line command is not what decides authentication. The method list is.

TACACS+ versus RADIUS

TACACS+ RADIUS
Transport TCP 49 UDP 1812 (auth) and 1813 (accounting); legacy 1645 and 1646
Encryption Whole packet body Only the User-Password attribute
AAA separation Authentication, authorization and accounting are separate exchanges Authentication and authorization are combined in Access-Accept
Per-command authorization Yes No practical equivalent
Typical use Device administration (who can type what on a router) Network access: 802.1X, VPN, wireless, and also device login
Standard Cisco-originated, documented in RFC 8907 RFC 2865 and RFC 2866

On IOS the radius server definition has historically defaulted to ports 1645 and 1646. A modern server such as Cisco ISE or FreeRADIUS listens on 1812 and 1813. Always set the ports explicitly with auth-port 1812 acct-port 1813 and confirm them in show aaa servers. A port mismatch looks exactly like a dead server.

Method lists and the ERROR versus FAIL rule

A method list is an ordered list of sources. IOS tries the first one and moves to the next only when the current method returns ERROR, which means it could not give an answer: no server responded, every server in the group timed out, or the connection was refused. If a method returns FAIL, which means a server answered and rejected the credentials, the process stops and the user is denied.

Method keyword Meaning
group tacacs+ All configured TACACS+ servers, in the order they were defined
group radius All configured RADIUS servers
group NAME Only the servers in that server group
local Local username database, username not case sensitive
local-case Local username database, username case sensitive
enable The enable secret
line The line password
none No authentication, always succeeds
if-authenticated (Authorization only) permit if the user was authenticated 
graph TB
  A["Method 1: group TAC-GRP"] --> B{"Did any server answer?"}
  B -->|"No: timeout, refused, unreachable = ERROR"| C["Method 2: local"]
  B -->|"Yes, PASS"| D["Access granted"]
  B -->|"Yes, FAIL"| E["Access denied, method 2 never tried"]
  C --> F{"Local user and password match?"}
  F -->|"yes"| D
  F -->|"no"| E

The consequences are what the exam tests:

  1. A user who exists only in the local database cannot log in while the TACACS+ server is up. The server rejects the unknown user with FAIL.
  2. A method list of group tacacs+ with nothing after it locks everyone out when the server is unreachable. There is no next method to try.
  3. When the server is unreachable, every login waits for the server timeout (5 seconds by default for each server in the group) before falling back. Slow logins that eventually work are a strong hint that the server is not answering.

Exam trap

"The local account works when the server is down but not when it is up" is normal behaviour, not a fault. It is FAIL ending the list. If the requirement says the local account must always work, the only clean fixes are to add the user on the server or to put local first, which changes who is authoritative.

Configuration reference

username netadmin privilege 15 secret Str0ng-Local-1
aaa new-model
!
tacacs server TAC1
 address ipv4 10.50.1.10
 key Tac-Key-01
 timeout 3
tacacs server TAC2
 address ipv4 10.50.1.11
 key Tac-Key-01
!
aaa group server tacacs+ TAC-GRP
 server name TAC1
 server name TAC2
 ip tacacs source-interface Loopback0
!
aaa authentication login VTY group TAC-GRP local
aaa authentication login CONSOLE local
aaa authorization exec VTY group TAC-GRP local
aaa authorization commands 15 VTY group TAC-GRP local
aaa accounting exec VTY start-stop group TAC-GRP
aaa accounting commands 15 VTY start-stop group TAC-GRP
!
line con 0
 login authentication CONSOLE
line vty 0 15
 login authentication VTY
 authorization exec VTY
 authorization commands 15 VTY
 accounting exec VTY
 accounting commands 15 VTY
 transport input ssh

username netadmin privilege 15 secret Str0ng-Local-1
aaa new-model
!
radius server RAD1
 address ipv4 10.50.1.20 auth-port 1812 acct-port 1813
 key Rad-Key-01
!
aaa group server radius RAD-GRP
 server name RAD1
 ip radius source-interface Loopback0
!
aaa authentication login VTY group RAD-GRP local
aaa authorization exec VTY group RAD-GRP local
aaa accounting exec VTY start-stop group RAD-GRP
!
line vty 0 15
 login authentication VTY
 authorization exec VTY
 accounting exec VTY
 transport input ssh

username netadmin privilege 15 secret Str0ng-Local-1
username helpdesk privilege 5 secret Help-Desk-1
aaa new-model
aaa authentication login default local
aaa authorization exec default local
!
line vty 0 15
 transport input ssh

A few rules worth memorising:

  • Named lists must be attached. aaa authentication login VTY ... does nothing on a line until login authentication VTY is configured there. The same goes for authorization exec VTY and accounting exec VTY under the line. A default list is used automatically.
  • The privilege level comes from EXEC authorization. A server can return priv-lvl=15 (TACACS+) or shell:priv-lvl=15 in a Cisco AV pair (RADIUS), but IOS only asks for it if an aaa authorization exec list applies to the line. Without one, the user starts at privilege level 1.
  • Command authorization skips configuration mode commands unless aaa authorization config-commands is configured.
  • The console is exempt from authorization unless aaa authorization console is configured. Authentication on the console still uses the default list if no named list is attached.
  • Source interface matters. The server checks the source address of each request against its list of network devices. By default IOS uses the outgoing interface address. Set ip tacacs source-interface or ip radius source-interface, either globally or inside the server group.

Exam trap

A console with no named list uses the default login list. If the default list is group tacacs+ with no fallback and the server is down, the console is locked out too. A dedicated console list (local, or local none in a lab) is the standard safeguard.

Verification and diagnostic commands

Command What it proves
show aaa servers Each server's state (UP or DEAD), ports, request, timeout and error counters
show tacacs TACACS+ socket opens, failed connect attempts, packets sent and received
show aaa method-lists all Every method list and the methods in it
show aaa sessions and show users Who is logged in, on which line
show running-config \| section line vty Which named lists the lines actually use
test aaa group TAC-GRP netadmin Pass1 legacy Sends one test authentication to the group without logging in
debug aaa authentication Which method list was picked and what each method returned
debug aaa authorization EXEC and command authorization requests and replies
debug tacacs or debug radius Protocol-level detail, including key and packet errors

Here is a healthy show aaa servers entry for comparison with the faults below:

R1# show aaa servers
TACACS+: id 1, priority 1, host 10.50.1.10, auth-port 49
     State: current UP, duration 8611s, previous duration 0s
     Dead: total time 0s, count 0
     Authen: request 14, timeouts 0, failover 0, retransmission 0
             Response: accept 12, reject 2, ...

A useful debug sequence on a failed SSH login shows which list was picked and what the server answered:

R1# debug aaa authentication
AAA/BIND(0000001A): Bind i/f
AAA/AUTHEN/LOGIN (0000001A): Pick method list 'VTY'
TPLUS: Queuing AAA Authentication request 26 for processing
TPLUS: Authentication start packet created for 26(netadmin)
TPLUS: Using server 10.50.1.10
...
TPLUS: Received authen response status FAIL

The output above is abridged and its exact wording varies by release, but the two lines that matter are always there: the method list that was picked and the status the server returned. Pick method list 'default' when you expected 'VTY' means the named list is not attached to the line. A reply of FAIL means the server rejected the user, so the local method will not be tried.

Fault table

Symptom Likely cause Proof
Every login waits about 5 seconds, then the local password works Server unreachable, wrong key with TACACS+ connection closed, wrong source address, or wrong port show aaa servers shows timeouts or DEAD; show tacacs shows failed connect attempts
Local account rejected, server account works Normal FAIL behaviour; user is unknown on the server debug aaa authentication shows FAIL from the server
All logins fail when the server is down Method list has no fallback (group tacacs+ only) show aaa method-lists authentication
Server account logs in at R1> instead of R1# No aaa authorization exec list on the line, or server does not return the privilege attribute show privilege; show run \| include authorization exec
Login succeeds, then the session drops immediately EXEC authorization fails (server down and no fallback, or server denies the shell service) debug aaa authorization shows FAIL for exec
Show commands work, configure terminal is denied Command authorization at level 15 denies it on the server debug aaa authorization per command
Server log says "unknown device" Request sourced from the wrong interface Server log; fix with ip tacacs source-interface
debug tacacs reports an invalid or undecryptable packet Shared key mismatch Compare the key on both sides; retype it, watch for trailing spaces
Named list changed, behaviour did not List not attached under the line; default list still in use debug aaa authentication shows Pick method list 'default'

Exam trap

A key mismatch does not always look like a reject. With TACACS+ the server typically cannot decrypt the request and closes the connection, which IOS treats as an ERROR and falls back to the next method. The user may log in with the local account and never notice that the server is not being used. Watch the server counters, not just whether login worked.

Lab: AAA break/fix with RADIUS

This lab uses RADIUS because FreeRADIUS is a standard Ubuntu package and is easy to run in CML. The troubleshooting logic is the same for TACACS+.

Topology

graph LR
  H["Admin host<br/>192.168.100.10"] -- "192.168.100.0/24" --- R2
  SRV["FreeRADIUS Ubuntu<br/>192.168.100.50"] -- "192.168.100.0/24" --- R2
  R2 -- "Gi0/0 .2  10.1.2.0/30  .1 Gi0/0" --- R1["R1<br/>Lo0 10.255.255.1"]
Device Interface Address
R1 Gi0/0 10.1.2.1/30
R1 Loopback0 10.255.255.1/32
R2 Gi0/0 10.1.2.2/30
R2 Gi0/1 192.168.100.1/24
Ubuntu server eth0 192.168.100.50/24, gateway .1
Admin host eth0 192.168.100.10/24, gateway .1

Run OSPF area 0 on R1 and R2 for all of these networks, or use static routes, so that R1's Loopback0 and the management LAN reach each other.

Steps

  1. On the Ubuntu server, install FreeRADIUS (sudo apt install freeradius). Add R1 as a client in /etc/freeradius/3.0/clients.conf:

    client R1 {
    ipaddr = 10.255.255.1
    secret = Rad-Key-01
}

  2. Add a user at the top of /etc/freeradius/3.0/users, then run the server in the foreground with sudo freeradius -X so you can watch requests:

    netops  Cleartext-Password := "Rad-User-1"
        Service-Type = NAS-Prompt-User,
        Cisco-AVPair = "shell:priv-lvl=15"

  3. On R1, build SSH and the local fallback user first, then AAA:

    hostname R1
ip domain name lab.local
crypto key generate rsa modulus 2048
ip ssh version 2
username netadmin privilege 15 secret Local-Pass-1
aaa new-model
radius server RAD1
 address ipv4 192.168.100.50 auth-port 1812 acct-port 1813
 key Rad-Key-01
aaa group server radius RAD-GRP
 server name RAD1
 ip radius source-interface Loopback0
aaa authentication login VTY group RAD-GRP local
aaa authentication login CONSOLE local
aaa authorization exec VTY group RAD-GRP local
line con 0
 login authentication CONSOLE
line vty 0 4
 login authentication VTY
 authorization exec VTY
 transport input ssh

  4. Verify the baseline. Run test aaa group RAD-GRP netops Rad-User-1 legacy and expect User was successfully authenticated. SSH from the admin host as netops and confirm the prompt is R1#. Save with copy running-config flash:aaa-baseline.cfg.

  5. Break 1: remove ip radius source-interface Loopback0 from the group. Log in as netops. Expected symptom: a pause, then the login fails for netops but netadmin with the local password works after the pause. freeradius -X shows a request from an unknown client 10.1.2.1. show aaa servers shows timeouts climbing. Fix by restoring the source interface.

  6. Break 2: under line vty 0 4 enter no authorization exec VTY. Log in as netops. Expected symptom: login succeeds but the prompt is R1>. Note that default exec authorization does not exist, so nothing asks the server for the privilege level. Fix by reattaching the list.

  7. Break 3: change the method list to aaa authentication login VTY group RAD-GRP (no local), then shut down the server (sudo systemctl stop freeradius, or stop the foreground process). Expected symptom: no one can log in over SSH. The console still works because it has its own list. Fix by restoring local as the second method.

  8. Break 4: change the key on R1 to Rad-Key-02. Watch debug radius on R1 and freeradius -X on the server during a login. Note what each side reports and whether the login falls back to the local account. Restore the key.

Expected verification output after the fixes

R1# show aaa servers | include RADIUS|State|Authen
RADIUS: id 1, priority 1, host 192.168.100.50, auth-port 1812, acct-port 1813
     State: current UP, duration 312s, previous duration 0s
     Authen: request 6, timeouts 0, failover 0, retransmission 0

R1# show privilege
Current privilege level is 15

Quick check

A router has aaa authentication login default group tacacs+ local. The TACACS+ server is up and rejects the password for user backup, which exists only in the local database. Can backup log in?

No. The server answered with FAIL, which ends the method list. local is tried only when TACACS+ returns ERROR, for example when no server responds.

TACACS+ users log in successfully but always land at privilege level 1, even though the server returns priv-lvl=15. What is the most likely missing command?

An EXEC authorization method list that applies to the VTY lines, such as aaa authorization exec default group tacacs+ local, or a named list attached with authorization exec NAME under the line. The privilege level is delivered as an EXEC authorization attribute.

Why should RADIUS ports be set explicitly in the radius server block?

IOS has historically defaulted to the legacy ports 1645 and 1646, while current servers listen on 1812 and 1813. A port mismatch makes the server look dead, and logins fall back to the next method after a timeout.

References