How netviz is put together

This is the orientation note for somebody about to change the code. It says what the pipeline is, which module owns each stage, what each stage may assume about its input and what it promises the next one — so that a change can be made in one place instead of five. For how to build, test and submit it, see CONTRIBUTING.md.

Contents

• The pipeline
• Stage 1: load
• Stage 2: validate
• Stage 3: build the graph
• Stage 4: narrow and summarise
• Stage 5: render
• The side branches
• Cross-cutting pieces
• Module map
• Using it as a library
• Design rules a change must not break
• See also

The pipeline

Five stages, each a plain function, each independently testable. The type on each arrow is the whole contract between one stage and the next.

     Path
      │   loader.load_tree / load_stream
      │   walk, parse, expand ranges, merge templates, index by qualified name
      ▼
  Inventory ───────► validate.validate ───────► list[Finding]
      │                                              │
      │                                              ▼
      │                                   diagnostics.build_report
      │                               (text | json | sarif | github)
      │   render.graph.build_graph
      │   resolve names, VLANs, tunnels, panels, prefixes — exactly once
      ▼
    Graph
      │   render.graph.filter_graph          remove, …
      │   render.aggregate.aggregate_graph   … then summarise, in that order
      ▼
    Graph
      │   render.render(graph, format, options)
      ▼
  str | bytes

cli._build_graph is the only place the CLI knows this order; watch.pipeline.run_cycle and web.preview.render_source walk the same five stages for their own front ends. Nothing else re-implements it.

Stage 1: load

Owner: src/netviz/loader/ — tree.py (the walk), documents.py (the strict YAML parser), inventory.py (the index), cache.py (parsed files, remembered by content), plus ranges.py, templates.py, provenance.py, ignore.py.

def load_tree(
    root: Path, *, keep_provenance: bool = False, cache: DocumentCache | None = None
) -> Inventory: ...
def load_stream(
    text: str, *, name: str = STREAM_NAME, keep_provenance: bool = False
) -> Inventory: ...

cache makes a repeated load incremental without making it stateful: every file is still read and hashed on every load, so the tree on disk remains the only thing that decides the result, and what is skipped is turning bytes that have been seen before back into elements. A hit is indistinguishable from a parse — same elements, same diagnostics, same order — which is what tests/test_cache.py asserts over every committed example. See docs/configuration.md.

May assume nothing. root is checked, and a path that does not exist or is neither a directory nor a YAML file is the only thing that raises (LoaderError). Everything a user can get wrong inside the tree is data.

Guarantees to every later stage. Every element is a validated model, parsed against the pydantic model of its kind before it is indexed, so a rule, a graph builder or a renderer may rely on field types, bounds and enum membership without re-checking them; and it is qualified, indexed under its directory plus metadata.name, so sites/berlin/rack1/sw1.yaml becomes sites/berlin/rack1/sw1, with lookups going namespace-first then global. Shorthands are already gone: interfaces[].range is expanded and spec.from merged against its template during the load, so no later stage re-parses YAML or re-expands anything — an inventory written with templates renders byte for byte like the same inventory written out longhand. Order is deterministic — files in byte-wise order of their relative POSIX path, documents in file order (NV-L005) — which is what makes every rendering and every golden file stable. And loading is total, a syntax error, schema violation, unreadable file or duplicate name being recorded on inventory.errors as a LoadError while the walk continues, and safe, because only StrictSafeLoader is used, so a hostile document cannot construct arbitrary Python objects.

keep_provenance=True additionally records which file, line and column each field came from, so a semantic finding can be reported at the line that caused it. It is off by default because the redirect tables hold the YAML node trees alive — 18 MB retained instead of 5 MB on a 628-element tree — and only the machine-readable validate formats need it.

Stage 2: validate

Owner: src/netviz/validate.py, with the catalogue in rules.py and the suppression settings in config.py.

def validate(inventory: Inventory, config: ValidationConfig | None = None) -> list[Finding]: ...

May assume the loader's guarantees: every document parses and matches the schema of its own kind, so a check never re-validates a field. What it may not assume is that the documents agree with each other — that is the question this stage asks.

Guarantees. It never raises for an inventory problem; it reports, and the caller decides. It never mutates the inventory — nothing in validate.py calls Inventory.add or Inventory.record, and tests/test_properties.py::test_validate_is_a_function_of_the_inventory_alone pins it. One problem is one finding: a duplicate address shared by five interfaces is a single finding naming all five, anchored at the first declaration in load order. Order is stable — source file, then position in the file, then severity and rule id. And suppression is a filter rather than a branch: the engine skips disabled rules and drops annotated findings afterwards, so a check cannot behave differently when a rule is re-graded.

Each check is a generator Callable[[_Context], Iterator[_Draft]] registered in _CHECKS, a tuple of (rule_id, check) pairs whose order must equal rules.RULE_IDS. _build_context resolves every cable endpoint, tunnel end, attachment and placement once, so a check is a loop over resolved records rather than a second resolver.

Stage 3: build the graph

Owner: src/netviz/render/graph.py.

def build_graph(inventory: Inventory, *, layer: Layer = Layer.L1) -> Graph: ...

May assume an Inventory — but not that validate has run: a cable whose endpoint does not resolve is dropped and recorded on Graph.dangling, because a half-written inventory still has to draw.

Guarantees. Graph is frozen: nodes a mapping in load order, edges a tuple, and every edge references two nodes that exist, so a renderer never has to check. Resolution happens exactly once, here — name resolution, VLAN membership, adapter attachment, patch-panel splicing, tunnel and encapsulation stacks and layer-3 prefix derivation are all this one pass, which is why a diagram, a traced path, an export and a networkx view cannot disagree about what is connected to what. Graph.sources says where each element was written, for --link-template, and is deliberately not part of what a renderer draws.

Layer decides what the graph is: physical keeps patch panels and their segments, l1 splices each run into one edge, l2 annotates that with VLAN membership, l3 replaces cables with prefixes as nodes, overlay makes tunnels nodes, and rack builds no topology at all — one node per rack holding its elevation. rack_elevations, resolve_tunnels and splice_patch_panels expose parts of the same pass.

Stage 4: narrow and summarise

Owners: render/graph.py (filtering) and render/aggregate.py (aggregation).

def filter_graph(graph: Graph, spec: FilterSpec) -> Graph: ...
def aggregate_graph(graph: Graph, spec: AggregateSpec | None = None) -> Graph: ...

May assume a Graph from stage 3, or from itself — both are Graph → Graph, so they compose. Guarantees a Graph with the same invariants. The two run in this order and only this order, and cli._build_graph says why: filtering decides what exists, aggregation folds what is left. Reversed, --kind switch could empty a collapsed node of everything it claims to stand for.

FilterSpec removes: namespaces, vlans, kinds, names, neighbors_of/depth and selected. Values within one field are alternatives, different fields are combined with AND, and every field selects elements — so a derived layer-3 subnet node survives exactly as long as one selected element still has an address in it.

selected is the selector language, layered over this stage rather than implemented inside it. A query can traverse the graph and can ask about interfaces, so it must see the whole graph and needs a parser and a vocabulary the renderer has no business knowing about; what arrives in selected is therefore the answer — a set of fully-qualified names — computed by netviz.query against the unfiltered graph. query/apply.py is the eighty lines that do the two in the right order, and filter_graph raises rather than proceeds if it is handed a select nobody answered, because rendering the whole inventory for a query that selects three devices is a silently wrong picture. AggregateSpec removes nothing: collapse and collapse_depth replace a namespace with one node that says which elements it stands for, and bundle folds parallel links into one edge carrying the count. aggregate_graph returns graph itself when nothing applies, so a pipeline that never aggregates is byte-identical.

Stage 5: render

Owner: src/netviz/render/ — the registry in registry.py, the backends in dot.py, html.py, mermaid.py and jsonexport.py, and the shared display decisions in options.py, details.py, ids.py, links.py, icons.py, highlight.py, fragment.py.

def render(graph: Graph, format: str, options: RenderOptions | None = None) -> bytes: ...
def render_text(graph: Graph, format: str, options: RenderOptions | None = None) -> str: ...
def render_layers(
    graphs: Sequence[Graph], format: str, options: RenderOptions | None = None
) -> bytes: ...

May assume a Graph and a RenderOptions, and nothing else: a backend never sees the Inventory, never reads netviz.toml, and never decides what exists.

The stored arrangement (§18) reaches a backend on the Graph, not in the options, and that placement is the point: geometry is a fact about the inventory rather than a preference about the drawing, so svg, html and json agree on coordinates without any of them being told to and none of them can be told something different. build_graph resolves it, filter_graph narrows it to what survived, and render/dot.py turns it into pos attributes and a Graphviz command line — see layout/geometry.py for the three cases.

Guarantees. render always returns bytes — text formats UTF-8 encoded, image formats as Graphviz produced them — so a caller writing to a file or stdout needs no per-format branching; render_text is for callers that want a string and have already excluded the binary formats. Output is byte-for-byte stable for a given graph and options, which tests/test_golden.py asserts and which is what makes netviz render -f dot > topology.dot a file worth committing.

RENDERERS in registry.py is the single declaration of a format. Every fact a front end could want — suffix, media type, whether the output is binary, whether it can draw icons, a highlight, a rack elevation or several layers at once, what content-security policy it needs, what to warn about at a given graph size — is a field on its Renderer. No front end branches on a format name: -f's choices, the help text, the preview server's content type and the size advisories all derive from that mapping.

The side branches

Everything else hangs off one of the five stages and adds no sixth.

Four are worth a sentence more. netviz.graph sits beside the renderers, not under them: it hands the graph build_graph already resolved to networkx so that connectivity questions are answered by graph algorithms, and it declares its own filter_graph — over nx.MultiGraph, with keyword predicates rather than a FilterSpec — so check which one an import means. fmt/ is the one part that does not share a parser with the rest: ruamel.yaml's round-trip loader keeps the comments, blank lines and quoting style a formatter cannot discard, every formatted document is handed back to the strict loader before it is written (fmt/verify.py), and nothing on the loading path imports it, so validate pays nothing for it. edit/ is the mirror of fmt/ and shares its parser for the same reason, but not its remit: fmt rewrites whole files and edit re-emits only the documents an operation named, so that every other byte of a file survives an edit. It is the one branch that runs two stages of the pipeline as a check — it loads and validates the tree it would write, through loader.Overlay, and refuses if the edit would introduce a new error. And export/ is scoped like a render on purpose: the same FilterSpec narrows both, so --namespace sites/north --kind switch means the same thing to a diagram and to a hosts file, and each emitter records what it had to drop (export/manifest.py) instead of dropping it quietly.

Cross-cutting pieces

errors.py and diagnostics.py — diagnostics. errors.py holds the exception hierarchy: NetvizError and its subclasses ConfigurationError, LoaderError (with SchemaError/SchemaIssue), ValidationError and RenderError. The CLI catches the base class, so a new failure mode gets a clean message by inheriting from it rather than by adding a try in cli.py. It also owns the text helpers every diagnostic uses — count_text, clip_text, echo_value, compact_ids — which is why no message concatenates an unbounded value into itself. diagnostics.py turns load errors and findings into one sorted stream of Diagnostic records and serialises it as JSON, SARIF 2.1.0 or GitHub workflow commands; its LOAD_RULE pseudo-rule gives a load error a rule id and a documented section too, so no diagnostic reaches a user without one.

settings.py and config.py — netviz.toml to Click parameters. config.py reads the file: [validate] (ignore, severity, strict) and, handing them straight over, [render] and every [profile.<name>] block. Unknown keys inside a known table are rejected — a misspelt ingore = [...] that silently did nothing would be worse than a failed run — while unknown top-level tables are left alone so a file shared with a later version still works. settings.py owns the one naming rule (a key is the long flag without its leading dashes), the SETTINGS registry mapping each key to a Click parameter and a parser, and the precedence ladder in resolve_settings: explicit flag, then the selected profile, then [render], then the Click default. Each result is a Resolution carrying its Origin, which netviz config show prints as provenance.

rules.py — the single rule catalogue. Every rule the validator can report is declared here exactly once, with a permanent short id (E###, W###, I###), a default severity, a one-line summary, its NV-* schema aliases and the title that Rule.anchor and Rule.help_uri derive a deep link into validation-rules.md from. Ids are permanent: once assigned, one is never reused, so a suppression in somebody's inventory keeps meaning what it meant. Keeping the catalogue in its own module is what lets config.py resolve and check rule ids without importing the validator.

Three smaller ones. console.py holds tables, colour and TTY detection, and a Console is handed to a command rather than constructed by it, so --quiet, --verbose and --color are honoured without each command re-deciding. httpserve.py holds what the two local servers promise — loopback binding, the default content-security policy, security headers, a Host check — because watch and web are different applications but what they promise about being a local server has to be identical.

fsio.py holds the three questions about writing a file that must have the same answer everywhere netviz runs: what a line ending is (\n, never the platform's, because a canonical form and a golden file are defined in bytes), how a file is replaced (through a sibling temporary and os.replace, retried for the sharing violation only Windows raises), and what a generated file may be called (not nul.yaml, which is a device on Windows and not a file). Each of those had two implementations and one of them was missing the newline argument, which is the shape of bug this module exists to make impossible: tests/test_platform.py fails if any call site goes back to Path.write_text.

Module map

Verified against the tree: every path below exists.

Using it as a library

The package is typed (py.typed) and checked with mypy --strict, so the stages above are usable directly. Nothing in them needs the CLI.

from pathlib import Path

from netviz.config import load_config
from netviz.loader import load_tree
from netviz.render import RenderOptions, build_graph, icon_theme, render
from netviz.validate import validate

root = Path("inventory")
inventory = load_tree(root)
for finding in validate(inventory, load_config(root).validation):
    print(finding)  # inventory/sw1.yaml#0:3: error: E002: ...

options = RenderOptions(show_ips=False, icons=icon_theme("cisco"))
svg = render(build_graph(inventory), "svg", options)

load_tree never raises for a problem inside the tree; unreadable documents are collected on inventory.errors. validate never raises either. Text that never was a folder — a paste, a pipe, a snippet from a ticket — goes through load_stream instead, and render_source runs the whole of what netviz web does per keystroke in one call:

from netviz.loader import load_stream
from netviz.render import Layer
from netviz.web import ViewOptions, render_source

text = Path("topology.yaml").read_text()
inventory = load_stream(text)  # same parser, same schema, same rules

preview = render_source(text, ViewOptions(layer=Layer.L2))
preview.svg  # an <svg> fragment, safe to embed, with an id on every element
preview.details["node-sw-office"]  # the info box, keyed by the drawn element's id
preview.problems  # load errors and findings, most severe first

The keys of preview.details are the ids render/ids.py gives the drawn elements — node- or edge- followed by a slug of the fully-qualified name, so node-sw-office and edge-cables_cbl-rtr-sw. They are the same ids -f svg and -f json carry, which keeps a hover, a tooltip and an export from drifting apart. And render_source never raises for anything the text can be wrong about: a syntax error, a dangling cable and a filter that matches nothing all come back as a preview whose status and problems say so, with whatever resolved still drawn.

Design rules a change must not break

Each is load-bearing somewhere else in the tree, and a test names most of them.

1. Models forbid unknown keys. NetvizModel sets extra="forbid" (models/base.py, NV-D005): silently ignoring a misspelt key would produce a diagram that disagrees with the file, which is the failure mode this tool exists to prevent. It is also what puts additionalProperties: false into the JSON Schema.
2. The loader normalises so that no later stage re-parses. Ranges are expanded and templates merged during the load, and nothing below loader/ and fmt/ imports a YAML parser — cli.py uses yaml.safe_dump for output only — so an inventory cannot mean one thing to validate and another to render.
3. Validation never mutates the inventory. validate reads and returns list[Finding]; a check that wrote something down would be one whose result depended on run order.
4. Renderers are pure functions of (Graph, RenderOptions). A backend gets no Inventory, no configuration and no filter. RenderOptions says what to draw; FilterSpec decided what exists before any renderer ran — so turning a label off can never change the topology a reader sees.
5. Every user-visible diagnostic goes through rules.py, so it has a permanent id, a severity and a title from which its section in validation-rules.md is derived. Load errors are not exempt: report.LOAD_RULE gives them the same three. tests/test_docs.py fails when a rule has no section, a wrong severity, a missing alias or no "Suppress with" line, and tests/test_examples.py fails when it has no fixture under tests/fixtures/invalid/.
6. One entry per output format, in render/registry.py. No front end may branch on a format name; something new to know about a format is a new field on Renderer.
7. Output is deterministic, and identifiers are permanent. Load order, finding order, node and edge order and every export's collation are explicitly sorted, never left to dict order or directory traversal — tests/test_golden.py and the export fixtures would notice. And a rule id, a NodeType, a JSON export key or a drawn element's id appears in somebody's suppression list, style sheet or downstream tool: add, do not renumber.

See also

• CONTRIBUTING.md — the gates, the recipes, how to run CI's checks.
• Testing netviz — the example half and the property half of the suite, and the Hypothesis profiles.
• The inventory schema — the specification the models and the loader implement, section by section.
• Validation rules — the write-up behind every id in rules.py.