Skip to content
Log in Sign up My account

5.3 and 5.4 Model-Driven Telemetry, gRPC and gNMI

Blueprint 5.3: Impact of model-driven telemetry on the network

Understand what streaming telemetry costs the device, the network and the collector, and how to size and limit it.

Blueprint 5.4: Model-driven telemetry with gRPC and gNMI (dial-in and dial-out)

Choose dial-in or dial-out, and periodic or on-change publication, for a given requirement. 5.4.a periodic publication. 5.4.b on-change publication.

ENCOR touches telemetry only as an input to Catalyst Center Assurance (see Catalyst Center) and the model foundation in YANG data modeling. ENSLD asks the design questions: what does streaming do to my network, and which subscription style fits.

Why telemetry replaces polling

SNMP polling Model-driven telemetry
Direction Pull: the manager asks Push: the device streams, or the collector subscribes once
Granularity Typically 1 to 5 minutes, because every poll costs a request and a response Seconds, or immediately on change
Data structure MIB OIDs, flat YANG-modelled, hierarchical, self-describing
Encoding BER GPB (protobuf), JSON, XML depending on protocol
Device cost Poll bursts, snmpd process load, full table walks One subscription, data read per interval, can be filtered by XPath
Event detection Wait for the next poll, or rely on traps On-change publication sends the event as it happens
Transport UDP TCP or HTTP/2 (reliable)

The design benefit is fidelity and timeliness. The design cost is that you now run a streaming pipeline: collector, time-series database and dashboards, and you must size it.

How a subscription works

graph LR
  subgraph D ["Device"]
    DS["YANG operational data<br/>(XPath)"] --> SUB["Subscription<br/>filter + period or on-change"]
  end
  SUB -->|"gRPC / HTTP/2<br/>protobuf or JSON"| COL["Collector"]
  COL --> TSDB["Time-series DB"]
  TSDB --> DASH["Dashboards and alerts"]

A subscription has four parts: what to send (an XPath or YANG path), how often (a period, or on change), how to encode it, and who receives it.

Dial-in and dial-out (5.4)

Dial-in Dial-out
Who opens the connection The collector connects to the device The device connects to the collector
Where the subscription is created By the collector at run time over gNMI (or NETCONF dynamic subscription) In the device configuration
Subscription lifetime Tied to the session. If the session drops, the subscription ends Persistent in config. The device reconnects after a failure
Firewall and NAT Needs inbound access to every device Outbound from device to a few collectors, usually simpler
Scale Collector manages many sessions and must re-subscribe Device-driven, scales to many devices behind NAT
Who changes the data set The collector, on the fly A configuration change on each device
IOS XE terminology Dynamic subscription Configured subscription
Typical use Tool that explores and tunes subscriptions interactively, or a controller that owns device state Production fleet monitoring at scale 
graph TB
  subgraph DI ["Dial-in"]
    C1["Collector"] -->|"1 connect, subscribe"| R1["Router"]
    R1 -->|"2 stream"| C1
  end
  subgraph DO ["Dial-out"]
    R2["Router<br/>(subscription in config)"] -->|"1 connect"| C2["Collector"]
    R2 -->|"2 stream"| C2
  end

gRPC and gNMI

  • gRPC is an RPC framework that runs over HTTP/2 and typically uses protocol buffers for encoding. HTTP/2 multiplexes streams over one TCP connection, which suits long-lived telemetry.
  • gNMI (gRPC Network Management Interface) is a service definition on top of gRPC, developed with the OpenConfig community. It has four RPCs: Capabilities (learn models and encodings), Get (one-time read), Set (change configuration) and Subscribe (telemetry).
  • gNMI is dial-in only by definition: the client connects to the device. Dial-out uses the vendor's gRPC dial-out service, which on IOS XE is configured under telemetry ioxe-mdt.
  • gNMI paths follow the model, so OpenConfig is the natural choice for a multi-vendor collector. Cisco operational models work well for platform-specific data.
  • Use TLS with certificates for production. Plain-text gRPC is for the lab only.

Subscription modes in gNMI

Mode Behaviour Fit
Stream, sample Send the value every N seconds Counters, utilisation
Stream, on-change Send only when the value changes, plus an initial full sync State: interface status, BGP neighbour state
Stream, target-defined Device picks sample or on-change per leaf Convenient default, less predictable
Once One snapshot, then close Inventory reads
Poll Collector triggers each update Rare

Periodic versus on-change (5.4.a and 5.4.b)

Periodic On-change
Sent when Every interval, always Only when a value changes (plus an initial sync)
Best for Counters and gauges that change constantly: bytes, CPU, queue depth Discrete state: link up/down, neighbour state, config change
Network cost Constant, proportional to paths and 1/interval Near zero while stable, bursty during incidents
Event detection Delayed up to one interval Immediate
Risk Over-subscribing a large table at a short interval A flapping object generates a flood. Not every leaf supports on-change
Gaps Missing sample means a lost sample Silence means no change, or a lost session; you need a heartbeat or periodic resync to tell the difference

On-change is not available for every leaf

Support for on-change depends on the platform and the specific path. Counters, for example, change constantly and are not suitable for on-change. A design must check which paths support it and fall back to periodic sampling for the rest.

On IOS XE, a configured period is expressed in centiseconds. update-policy periodic 1000 means every 10 seconds. A typical combination for one device is on-change for interface oper-status and BGP state, plus a 10 to 60 second periodic for counters.

telemetry ioxe-mdt subscription 101
 encoding encode-kvgpb
 filter xpath /interfaces-ios-xe-oper:interfaces/interface/statistics
 stream yang-push
 update-policy periodic 1000
 receiver ip address 10.99.0.50 57500 protocol grpc-tcp

R1# show telemetry ietf subscription all
R1# show telemetry internal connection

The first command lists configured and dynamic subscriptions and their state, and the second shows the connection to each receiver. State Valid on a dial-out subscription means the configuration is accepted, while the connection state shows whether the receiver is actually reachable.

Impact on the network and devices (5.3)

Streaming telemetry is cheap compared with polling, but not free. Design for each of the following.

Resource Impact Design response
Device CPU and memory Each subscription reads and encodes data on a schedule. Large tables (a full BGP RIB, a big MAC table) at short intervals are costly Filter with narrow XPaths, lengthen intervals, avoid whole-table subscriptions, limit the number of subscriptions per device
Control plane Telemetry processes compete with routing for CPU on small platforms Test on the smallest platform in the estate, watch CPU during bulk subscribe
Bandwidth Per device: payload size divided by period. Multiplied by thousands of devices at the collector Estimate: bytes per message times messages per second times devices. Use compact encodings (protobuf over JSON). Aggregate collectors regionally
WAN links Telemetry from branches can compete with business traffic on small circuits Send over the management path, mark and queue as management traffic (see QoS for network management traffic), use on-change where possible, and consider a regional collector
Collector Ingest rate, storage and query load grow with devices times paths times 1/interval Scale horizontally, set retention and downsampling, test with the production subscription count
Security The stream exposes topology and traffic metadata. Dial-in needs open management ports TLS, authentication, authorisation for telemetry users, management VRF, restrict receivers to known addresses
Failure behaviour If the collector is down, a dial-out device retries and may buffer or drop Redundant collectors, monitor the telemetry pipeline itself

Sizing sketch

A router publishes 40 interface counters in a 600-byte message every 10 seconds, which is about 60 bytes per second per device, or roughly 480 bits per second. For 5,000 devices that is about 2.4 Mbps at the collector, small as a bandwidth figure. The harder limit is usually the number of messages per second and the storage write rate, not the bits on the wire. Always compute both.

Subscription design checklist

  1. Name the question each path answers. If nobody uses a metric, do not subscribe to it.
  2. Use the narrowest XPath that returns what you need.
  3. Pick on-change for state, periodic for counters. Add a periodic resync if silence must be distinguishable from loss.
  4. Choose dial-out for the fleet and dial-in for interactive or controller-driven needs.
  5. Use TLS. Restrict receivers by address.
  6. Send telemetry over the management network or VRF, and classify it so it neither starves nor is starved by user traffic.
  7. Pilot on the smallest platform and measure CPU before rollout.

Exam trap

On-change does not mean "send everything when anything changes". It sends the specific leaf that changed. It also does not suit every counter. If a scenario wants per-interface byte counters graphed every few seconds, the answer is periodic, not on-change.

Exam trap

Dial-out is initiated by the device and the subscription is in its configuration. Dial-in is initiated by the collector and the subscription is created by the collector at run time. A "gNMI dial-out" answer is a distractor: gNMI clients dial in. Cisco IOS XE dial-out uses the telemetry configuration shown above.

Exam trap

Streaming telemetry reduces polling overhead but does not eliminate its own cost. "Telemetry has no effect on device CPU" is false, and a design that subscribes to full tables at one-second intervals can harm a small router.

Design exercise

Scenario. A bank runs 1,200 branch routers behind NAT and restrictive firewalls, each with a 20 Mbps WAN circuit shared with voice. The network operations team wants to detect link and BGP state changes within seconds and see interface utilisation trends. The collector pool sits in two data centres. Security does not allow inbound connections to branch devices. The routers have modest CPU.

Design the telemetry.

Model answer

Dial-out. Security prohibits inbound connections and the routers are behind NAT, so the collector cannot dial in. Configure dial-out subscriptions on each router, pointing at both data-centre collectors, with TLS and certificate authentication.

Mixed publication. Use on-change for interface oper-status and BGP neighbour state, which gives second-level detection with almost no steady-state traffic. Use periodic at a modest interval, for example 30 to 60 seconds, for interface byte counters, which only need to support trending. This keeps the per-branch stream to a few hundred bytes per second.

Protect the WAN and the CPU. Send telemetry in a management VRF or a defined management class, queued so it cannot compete with voice. Use narrow XPaths for statistics on the few interfaces that matter, avoid whole-table subscriptions such as the full RIB, and pilot on the smallest router first to confirm CPU headroom.

Rejected alternatives.

  • Dial-in gNMI: requires inbound access to 1,200 NATed devices and would be blocked by policy.
  • Periodic for everything at 1 to 5 seconds: detects events fast, but burns branch bandwidth and CPU and floods the collector with unchanged data.
  • On-change for everything: misses trend data, because counters either do not support on-change or flood the stream.
  • Stay with SNMP polling: multiplies request and response traffic over the thin WAN, gives minutes of granularity, and only offers traps for fast events.

Quick check

A security policy allows no inbound connections to devices, only outbound from the device network to a collector zone. Which style fits?

Dial-out. The device initiates the connection and the subscription lives in its configuration, so no inbound rule to the device is needed. Dial-in requires the collector to reach every device.

Which publication mode suits CPU utilisation, and which suits a BGP neighbour going down?

CPU utilisation changes constantly, so periodic sampling suits it. A BGP neighbour state change is a discrete event, so on-change gives immediate notification with almost no steady traffic.

Name two ways to limit the impact of telemetry on a small branch router.

Use narrow XPath filters instead of whole tables, lengthen sample intervals, use on-change for state, reduce the number of subscriptions, and pilot to measure CPU. Any two of these.

References