Skip to content
Log in Sign up My account

Git and CI/CD for Network Configs

Network changes are risky because a typo on one device can take down a site, and because nobody can say afterwards exactly what changed. Git and a CI/CD pipeline fix both problems. This article shows a repository layout, local checks with pre-commit, a pipeline that validates a change with Batfish (or pyATS) before it deploys, and how approvals and rollback fit in.

The idea

Treat the intended configuration as source code. Changes arrive as pull requests, tests run automatically, a person approves, and a pipeline deploys. The device is never the place where a change is first written.

graph LR
    ENG["Engineer edits<br/>branch"] --> PR["Pull request"]
    PR --> LINT["Lint and render"]
    LINT --> VAL["Batfish or pyATS<br/>validation"]
    VAL --> REV["Human review<br/>and approval"]
    REV --> MERGE["Merge to main"]
    MERGE --> DEP["Deploy to devices"]
    DEP --> POST["Post-change checks"]
    POST -->|fail| RB["Rollback"]

You do not need a large platform for this. A Git host with CI (GitHub, GitLab, Gitea with a runner), a runner that can reach your devices, and a few scripts are enough to start.

Repository layout

network-configs/
  README.md
  .pre-commit-config.yaml
  .github/workflows/validate.yml
  .github/workflows/deploy.yml
  inventory/
    hosts.yml
    group_vars/
  templates/
    access-switch.j2
    edge-router.j2
  vars/
    site-a.yml
    site-b.yml
  rendered/
    sw-a-1.cfg
    sw-a-2.cfg
  snapshots/
    candidate/
      configs/
  tests/
    test_batfish.py
  scripts/
    render.py
    deploy.py

Two common models exist, and you should pick one on purpose:

  • Templates plus variables (above): humans edit variables, a script renders configs with Jinja2, and rendered/ is committed so reviewers see the actual config diff. This scales better and avoids copy-paste drift.
  • Full configs in Git: each device's config is a file and people edit it directly. Simpler to start, harder to keep consistent.

Either way, commit the final per-device config text. Reviewers should be able to read the change as the device will see it.

Tip

One pull request per change, with a description that states the intent and the rollback plan. A pull request that touches 40 unrelated files cannot be reviewed.

Secrets do not belong in the repository. Use a secrets manager or the CI system's encrypted variables, and keep placeholders in the committed configs.

Pre-commit checks

pre-commit runs checks on your machine before a commit is created, so mistakes are caught in seconds. Install it with pip install pre-commit and pre-commit install. Using repo: local hooks avoids having to pin third-party repository versions:

repos:
  - repo: local
    hooks:
      - id: yamllint
        name: yamllint
        entry: yamllint
        language: system
        types: [yaml]

      - id: render-configs
        name: render configs from templates
        entry: python scripts/render.py --check
        language: system
        pass_filenames: false
        files: ^(templates|vars)/

      - id: no-secrets
        name: reject obvious secrets
        entry: python scripts/check_secrets.py
        language: system
        types: [text]

Sensible checks for a network repository:

  • yamllint for inventory and variable files.
  • A render check that fails if rendered/ is out of date with the templates and variables.
  • A secrets check. A simple script that rejects lines such as password 7, enable secret, snmp-server community with real values, or private keys is a good start. Dedicated scanners such as detect-secrets and gitleaks do this more thoroughly.
  • ansible-lint if you use Ansible, and terraform fmt -check and terraform validate if you use Terraform.

Local hooks can be bypassed, so run the same checks again in CI. Treat the pipeline as the enforcement point.

Validation with Batfish

Batfish is an open-source network configuration analysis tool. It parses device configs into a model and answers questions about them without touching a live device. It finds undefined references (a route-map applied but never defined), duplicate IPs, mismatched MTUs and BGP sessions that will not come up, and it can test reachability and ACL behaviour.

Run the service in a container:

docker run --rm -d --name batfish -p 8888:8888 -p 9997:9997 -p 9996:9996 batfish/batfish

Then install the Python client:

pip install pybatfish

Batfish expects a snapshot directory with a configs/ folder of device config files. Point the pipeline at your rendered/ output, copied to snapshots/candidate/configs/.

tests/test_batfish.py:

import pytest
from pybatfish.client.session import Session

@pytest.fixture(scope="module")
def bf():
    session = Session(host="localhost")
    session.set_network("prod")
    session.init_snapshot("snapshots/candidate", name="candidate", overwrite=True)
    return session

def test_no_init_issues(bf):
    issues = bf.q.initIssues().answer().frame()
    assert issues.empty, issues.to_string()

def test_no_undefined_references(bf):
    refs = bf.q.undefinedReferences().answer().frame()
    assert refs.empty, refs.to_string()

def test_bgp_sessions_compatible(bf):
    sessions = bf.q.bgpSessionCompatibility().answer().frame()
    bad = sessions[sessions["Configured_Status"] != "UNIQUE_MATCH"]
    assert bad.empty, bad.to_string()

These three tests catch a large share of real mistakes. Add more as you learn your network's failure modes: bf.q.traceroute(...) for reachability, bf.q.testFilters(...) to prove an ACL permits or blocks a flow, and bf.q.differentialReachability(...) with a reference snapshot to see what a change affects.

Question names and result column names can differ between Batfish versions. Print a frame once and check the column names before you assert on them.

Or validate with pyATS

pyATS tests a live (or lab) network rather than static configs. It can learn state before the change, apply it, learn again and diff. It suits post-change verification and CML or virtual lab runs in the pipeline, where you deploy to a replica first. Batfish validates intent offline and cheaply; pyATS validates behaviour on devices. Larger teams use both.

The pipeline

A validation workflow that runs on every pull request, in GitHub Actions syntax:

name: validate
on:
  pull_request:
    paths: ["templates/**", "vars/**", "rendered/**", "snapshots/**"]

jobs:
  validate:
    runs-on: ubuntu-latest
    services:
      batfish:
        image: batfish/batfish
        ports:
          - 9997:9997
          - 9996:9996
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Lint
        run: yamllint . && python scripts/render.py --check
      - name: Batfish tests
        run: pytest tests/ -v

Check the current major versions of those actions when you copy this. A matching deploy workflow runs only after merge and uses a protected environment:

name: deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: [self-hosted, network]
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Deploy changed devices
        env:
          NET_USER: ${{ secrets.NET_USER }}
          NET_PASS: ${{ secrets.NET_PASS }}
        run: python scripts/deploy.py --changed-since ${{ github.event.before }}

The self-hosted runner sits inside your network because hosted runners cannot reach your management addresses. Harden it: dedicated host, minimal access, no shared use, and restrict which repositories can use it.

scripts/deploy.py should do the same careful things as a human operator. It deploys only the devices whose files changed, pushes in small batches, saves a pre-change backup, and runs post-change checks. The push itself can use NAPALM, Netmiko or an Ansible playbook. Keep the deploy step thin; the intelligence belongs in the tests.

Approvals

Approvals come from three controls, used together:

  1. Branch protection on main: require a pull request, require the validation job to pass, and require at least one approving review. Disallow direct pushes, including for admins where your policy allows.
  2. CODEOWNERS so changes to core routers need approval from the team that owns them.
  3. A protected environment for the deploy job. In GitHub this is an environment with required reviewers, so the deploy waits for a named person to approve after merge. GitLab has protected environments and manual jobs for the same purpose.

This gives you two gates: the review of the change, and the decision to release it now. The second gate lets you merge during the day and deploy in the maintenance window.

Rollback

Plan rollback before you need it, at three levels.

1. Revert in Git. The cleanest rollback is git revert <commit>, a new pull request that undoes the change, which then flows through the same pipeline. It is auditable and repeatable. Use a fast-path approval process for reverts during incidents rather than skipping the pipeline.

2. Device-level rollback during the change. The deploy script takes a backup first and restores it if post-change checks fail. On IOS XE you can also use configure replace to a saved file, and reload in 15 before a risky change is a classic safety net you cancel with reload cancel once the change is verified. Use these mechanisms for a change that could cut off your own management access.

3. Post-change checks. A rollback needs a trigger. After deploying to each batch, check that management still works, that expected BGP and OSPF neighbours are established, and that key prefixes are present. If a check fails, stop the rollout and restore.

Warning

A config rollback restores configuration, not state. If a bad change reset BGP sessions or flushed a table, reverting the config does not bring back lost traffic. That is why batched deployment matters more than rollback speed.

Starting small

You do not need all of this on day one. A sensible order:

  1. Put device backups in Git on a schedule (the Ansible backup playbook does it).
  2. Add pull requests and a required review.
  3. Add linting and Batfish checks on the pull request.
  4. Add automated deploy to a lab or a low-risk site.
  5. Extend to production with protected environments and post-change checks.

Each step is valuable by itself. If you are studying for certification, the CCNP study plan pairs well with this: automating the configs makes the routing and switching fundamentals visible in diffs.

References