Skip to content
Log in Sign up My account

pyATS and Genie: snapshot, diff and test the network

pyATS is Cisco's open source Python test framework for network devices. Genie is the library on top of it that turns CLI output into structured data and models whole features such as OSPF or interfaces. This article shows how to write a testbed file, connect, parse and learn, take before and after snapshots around a change with the pyats command line, write a small AEtest script that fails when the network is not in the state you expect, and run it from CI.

What the pieces are

The names overlap, so it helps to separate them.

  • pyATS is the core: the testbed model, connection handling, the AEtest test harness, the Easypy job runner, logging and reporting.
  • Genie is the library of parsers (CLI text to Python dictionaries), models (learn, which runs several commands and merges them into one feature object), and the diff engine.
  • Unicon is the connection layer that pyATS uses to talk to devices over SSH or Telnet. You rarely call it directly.
graph LR
    TB["testbed.yaml"] --> C["Unicon connection"]
    C --> P["Genie parsers"]
    C --> L["Genie learn"]
    P --> A["AEtest checks"]
    L --> S["Snapshots"]
    S --> D["Diff before and after"]
    A --> R["Results and logs"]

Install it in a virtual environment. The full extra pulls in the parser library and the other components:

python3 -m venv .venv
source .venv/bin/activate
pip install "pyats[full]"

Check the supported Python versions for your pyATS release before you pick an interpreter. New pyATS releases usually trail the newest Python by a while.

The testbed file

A testbed is a YAML file that describes devices, how to reach them and how to log in. Keep secrets out of it. pyATS can read environment variables with the %ENV{NAME} syntax.

testbed:
  name: lab
  credentials:
    default:
      username: "%ENV{NET_USER}"
      password: "%ENV{NET_PASS}"
    enable:
      password: "%ENV{NET_ENABLE}"

devices:
  r1:
    os: iosxe
    type: router
    connections:
      cli:
        protocol: ssh
        ip: 192.0.2.11
        port: 22
        ssh_options: -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
  r2:
    os: iosxe
    type: router
    connections:
      cli:
        protocol: ssh
        ip: 192.0.2.12

The os key decides which parsers and which Unicon plugin are used, so it must be right. Common values are iosxe, ios, nxos, iosxr and junos. Use iosxe for Catalyst 9000 and ISR/ASR running IOS XE, and ios only for classic IOS.

Warning

The StrictHostKeyChecking=no option above is acceptable for a throwaway lab. In production, keep the host keys in a known hosts file and remove that option.

Validate the file and list what pyATS sees:

pyats validate testbed testbed.yaml

Connecting from Python

from genie.testbed import load

testbed = load("testbed.yaml")
r1 = testbed.devices["r1"]
r1.connect(log_stdout=False)

print(r1.execute("show clock"))
r1.disconnect()

load from genie.testbed builds the testbed and attaches the Genie parser and model mappings to each device. log_stdout=False stops the full session from being printed to the terminal. Other methods you will use:

  • device.execute("show ...") runs an exec command and returns the raw text.
  • device.configure("interface Gi1\n description test") enters config mode, sends the lines, and exits.
  • device.parse("show ...") runs a command and returns a dictionary.
  • device.learn("feature") returns a model object.

You can also connect to a pool of devices in parallel with testbed.connect(log_stdout=False).

Parsers

A Genie parser is bound to a command and an OS. The same parse call on an IOS XE device and an NX-OS device returns the same shape of data where the feature is the same, which is the point.

intf = r1.parse("show ip interface brief")

for name, data in intf["interface"].items():
    print(name, data["ip_address"], data["status"], data["protocol"])

From the command line you can parse without writing any Python:

pyats parse "show ip interface brief" \
    --testbed-file testbed.yaml \
    --devices r1 \
    --output parsed/

The output directory holds the parsed result and the raw connection log per device. If there is no parser for a command, parse raises a SchemaEmptyParserError or a "parser not found" error. Check the parser list on the Genie parser documentation site for your OS before you design around a command.

Tip

A parser raising SchemaEmptyParserError often means the command returned nothing, for example show ip ospf neighbor on a device with no OSPF neighbours. Treat that as an empty result in your own code if that is a valid state, and catch the exception.

Learn: whole-feature models

learn runs the set of commands that describe a feature, parses them, and merges the results into one object. The data lives in the info attribute.

ospf = r1.learn("ospf")
print(list(ospf.info["vrf"]))

Use learn when you care about the state of a feature as a whole. Use parse when you want one command and nothing else. Learn is slower because it runs several commands, and it is only available for the features that have a model for your OS. Genie documents the list of available features per platform in its model documentation.

Snapshot and diff around a change

This is the most practical use of Genie for day-to-day work. Take a snapshot before the change, make the change, take a second snapshot, and let Genie show only what differs.

# before
pyats learn ospf interface bgp \
    --testbed-file testbed.yaml \
    --devices r1 r2 \
    --output snap_before/

# ... make the change ...

# after
pyats learn ospf interface bgp \
    --testbed-file testbed.yaml \
    --devices r1 r2 \
    --output snap_after/

# compare the two directories
pyats diff snap_before/ snap_after/ --output diff_out/

Each snapshot directory contains one file per feature and device with the learned data, plus the connection logs. pyats diff compares the directories file by file and writes a diff file for each pair that differs. Open those files to see the added, removed and changed keys. If nothing changed for a feature, no diff file is produced for it.

The older genie learn and genie diff commands do the same thing and are still found in many tutorials. Prefer the pyats form in new work.

The same diff is available in Python for parsed output:

from genie.utils.diff import Diff

before = r1.parse("show ip interface brief")
# ... change ...
after = r1.parse("show ip interface brief")

diff = Diff(before, after)
diff.findDiff()
print(diff)

Warning

Counters, timers and uptimes change between any two snapshots. A raw diff of OSPF or BGP is full of noise unless you exclude those keys. Diff accepts an exclude list of key names, for example Diff(before, after, exclude=["up_time", "last_change"]). Build the exclude list the first time you run a real diff and keep it with the script.

A change window that uses this pattern is simple to describe:

graph TB
    A["Pre-checks pass"] --> B["learn: before"]
    B --> C["Apply change"]
    C --> D["learn: after"]
    D --> E["diff before and after"]
    E --> F{"Only expected changes?"}
    F -->|"Yes"| G["Close the change"]
    F -->|"No"| H["Roll back"]

A small AEtest script

AEtest is the structure for turning checks into a pass or fail result. A script has a common setup, any number of test cases, and a common cleanup. Each test case has optional setup, one or more test methods, and optional cleanup.

# checks.py
from pyats import aetest


class CommonSetup(aetest.CommonSetup):
    @aetest.subsection
    def connect(self, testbed):
        testbed.connect(log_stdout=False)
        self.parent.parameters["devices"] = list(testbed.devices.values())


class InterfacesUp(aetest.Testcase):
    """Uplinks must be up/up."""

    UPLINKS = {"r1": ["GigabitEthernet1"], "r2": ["GigabitEthernet1"]}

    @aetest.test
    def check_uplinks(self, testbed):
        problems = []
        for name, wanted in self.UPLINKS.items():
            dev = testbed.devices[name]
            out = dev.parse("show ip interface brief")["interface"]
            for ifname in wanted:
                data = out.get(ifname)
                if data is None:
                    problems.append(f"{name} {ifname} missing")
                elif data["status"] != "up" or data["protocol"] != "up":
                    problems.append(
                        f"{name} {ifname} {data['status']}/{data['protocol']}"
                    )
        if problems:
            self.failed("\n".join(problems))


class OspfNeighbors(aetest.Testcase):
    """Each router must have at least one OSPF neighbour in FULL."""

    @aetest.test
    def check_neighbors(self, testbed):
        for dev in testbed.devices.values():
            ospf = dev.learn("ospf")
            full = 0
            for vrf in ospf.info.get("vrf", {}).values():
                for af in vrf.get("address_family", {}).values():
                    for inst in af.get("instance", {}).values():
                        for area in inst.get("areas", {}).values():
                            for intf in area.get("interfaces", {}).values():
                                for nbr in intf.get("neighbors", {}).values():
                                    if str(nbr.get("state", "")).lower().startswith("full"):
                                        full += 1
            assert full > 0, f"{dev.name} has no FULL OSPF neighbour"


class CommonCleanup(aetest.CommonCleanup):
    @aetest.subsection
    def disconnect(self, testbed):
        testbed.disconnect()


if __name__ == "__main__":
    aetest.main()

Notes on the script:

  • Parameters such as testbed are passed into a section by name when the section declares an argument with that name. Easypy supplies testbed from the --testbed-file option.
  • self.failed("message") marks the test failed and stops it. A plain assert also fails the test with the message.
  • Because the OSPF model has a deep dictionary, print ospf.info once while developing and write the walk against the real data on your platform.
  • Use self.passed, self.failed, self.skipped and self.errored deliberately. errored means the test itself broke, which is different from the network failing a check.

To run it you need a job file. Easypy is the runner that handles the testbed, logs and archive:

# job.py
from pyats.easypy import run


def main(runtime):
    run(testscript="checks.py", runtime=runtime)

pyats run job job.py --testbed-file testbed.yaml

Results go to an archive directory. You can open the HTML log viewer for the last run with pyats logs view.

Running from CI

The script returns a non-zero exit status when any test fails or errors, so any CI system can gate on it. Here is a GitHub Actions job as an example. The credentials come from repository secrets, and the runner must have a network path to the devices (a self-hosted runner on a management network is the usual answer).

name: network-checks
on:
  schedule:
    - cron: "0 6 * * 1-5"
  workflow_dispatch:

jobs:
  checks:
    runs-on: [self-hosted, netmgmt]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install "pyats[full]"
      - run: pyats run job job.py --testbed-file testbed.yaml --archive-dir results
        env:
          NET_USER: ${{ secrets.NET_USER }}
          NET_PASS: ${{ secrets.NET_PASS }}
          NET_ENABLE: ${{ secrets.NET_ENABLE }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pyats-results
          path: results

The same two commands work in GitLab CI, Jenkins or a cron job. Run the checks after every automated change (see Git and CI/CD for network configs) and on a schedule, so drift shows up as a failed job and not as a ticket.

Gotchas

  • Prompts and terminal settings. Unicon recognises prompts by pattern. A custom banner, a modified prompt, or a pager that is not disabled can cause connection timeouts. Let Unicon set terminal length 0 itself and avoid exotic prompt strings.
  • Slow learn. A learn of BGP on a router with a large table can take a long time. Learn the features you need, on the devices you need, and use parse for narrow checks.
  • Missing parsers. Not every command has a parser on every OS. Check before you plan a check around one. A missing parser can be filled with your own regex on execute output.
  • Parser schema changes. Parser output keys can change between Genie releases. Pin the pyATS version in a requirements file and upgrade on purpose, then re-run your snapshots.
  • Noisy diffs. Excluding counters and timers is part of the job. Without it, the diff always shows changes and people stop reading it.
  • Config changes via configure. device.configure raises an error if the device reports an invalid command, which is useful, but it does not save the config. Save explicitly with device.execute("write memory") if you want that.
  • Shared testbeds. Several people running scripts against the same device at the same time will fight over VTY lines. Keep a limit on parallelism for older platforms.
  • Not a config-management tool. pyATS verifies state. Use Ansible, Netmiko or NAPALM to push the change, and pyATS to prove it worked.

Where it fits

A good first step is read-only: a testbed, a before and after learn around each change, and a short AEtest script of five or six checks (uplinks up, routing neighbours up, default route present). That alone catches most failed changes. Grow the checks from real incidents. If you want a lab to practise against, the setup notes at https://cisconets.com/ccnp/labs/ describe an environment you can point a testbed at.

References