zebra: EVPN VXLAN Multihome extern mode - Enable --kernel-mac-ext-learn

[pbrisset]

EVPN VXLAN Multihoming External Mode Support

Summary

This PR series implements complete EVPN VXLAN Multihoming External Mode support in zebra for platforms with Hardware-Based MAC Learning and Aging capabilities.

Motivation

Current EVPN VXLAN implementations rely on software-based MAC learning and aging in the kernel. However, modern data center switches with hardware acceleration capabilities can perform MAC learning and aging directly in the ASIC, which offers:

• Better performance: Hardware-based MAC learning/aging is faster and more efficient
• Reduced CPU overhead: Offloads MAC management from the CPU
• Scalability: Supports larger MAC tables with hardware resources

This PR enables FRR to work in cooperation with hardware-based MAC learning for EVPN VXLAN Multihoming scenarios.

Commit Overview

This PR contains 8 commits that progressively implement the feature:

1. Enable --kernel-mac-ext-learn - Infrastructure: command-line option and storage
2. New protocol RTPROT_HW - Add hardware-learned MAC protocol support
3. Documentation updates - Document --kernel-mac-ext-learn option
4. Debug print improvements - Enhanced debug output for external learn mode
5. ARP/ND suppression handling - Disable suppression in external learn mode
6. Netlink message handling - Handle MAC operations in external learn mode
7. MAC sync/update/delete/expiry - Complete MAC lifecycle management
8. Comprehensive topotests - Full test coverage for the feature
9. FPM-based test suite - Alternative test without kernel dependencies (~2513 lines) - NEW
10. Skip marker for kernel-dependent test - Disable original test until kernel patches merge
    Each commit compiles independently and adds a logical piece of the functionality.

Implementation Details

When --kernel-mac-ext-learn mode is enabled:

1. MAC entries are marked and programmed as extern_only in the kernel
2. Kernel aging is disabled for these MAC entries
3. Control plane: Zebra manages MAC lifecycle and BGP EVPN advertisements
4. Data plane: Hardware manages MAC learning, aging, and forwarding

Key Changes Across the Series

Commit 1: Infrastructure

Files: zebra/main.c, zebra/zebra_router.[ch], zebra/zebra_vxlan.h
Lines: +35, -15 (~50 lines changed)

• Added --kernel-mac-ext-learn command-line option
• Storage in zrouter.zav.kernel_mac_ext_learn
• Accessor function zebra_mac_ext_learn_mode()

Commit 2: Protocol Support

Files: zebra/zebra_dplane.h, zebra/rt_netlink.c
Lines: +1, -1 (~2 lines changed)

• New RTPROT_HW protocol for hardware-learned MACs
• Netlink integration for hardware protocol

Commit 3: Documentation

Files: doc/user/zebra.rst
Lines: +10 (~10 lines added)

• Comprehensive documentation of --kernel-mac-ext-learn option
• Usage examples and behavior description

Commit 4: Debug Enhancements

Files: zebra/zebra_evpn_mac.c
Lines: +7, -1 (~8 lines changed)

• Enhanced debug output for external learn mode
• Better visibility into MAC state transitions

Commit 5: ARP/ND Suppression

Files: zebra/zebra_vxlan.c
Lines: +11, -3 (~14 lines changed)

• Disable ARP/ND suppression when in external learn mode
• Proper handling of neighbor advertisements

Commit 6: Netlink Message Handling

Files: zebra/rt_netlink.c, zebra/zebra_dplane.c
Lines: +90, -39 (~129 lines changed)

• Handle MAC add/delete/update operations in external learn mode
• Proper extern_learn flag handling in netlink messages

Commit 7: MAC Lifecycle Management

Files: zebra/zebra_evpn_mac.c, zebra/zebra_evpn.c
Lines: +77, -28 (~105 lines changed)

• Complete MAC sync, update, delete, and expiry logic
• State machine for data plane ↔ control plane transitions
• Hardware aging coordination with zebra control

Commit 8: Kernel-Based Testing (0030)

Files: tests/topotests/bgp_evpn_mh_l2l3vni_ext_learn/, tests/topotests/lib/bgp_evpn.py
Lines: +1666, -48 (~1714 lines changed)
Status: ⏸️ Skipped until kernel patches merge (see Commit 10)

• New test suite: bgp_evpn_mh_l2l3vni_ext_learn (~933 lines)
• Validates ES discovery, VTEP peer lists, L2/L3 VNI operations
• Tests MAC protocol state transitions
• Covers orphaned, dual-attached, and single-attached host scenarios
• New utility library for EVPN testing (366 lines)
• Requires kernel RTPROT_HW support (not yet merged)

Commit 9: FPM-Based Test Suite (NEW)

Files: tests/topotests/bgp_evpn_mh_fpm_ext_learn/
Lines: +2513 (~2513 lines new)
Status: ✅ Fully functional, no kernel dependencies

• Alternative test suite using FPM (Forwarding Plane Manager)
• inject_mac.py: Python netlink tool simulating RTPROT_HW (322 lines)
• Main test file: 20 comprehensive tests (1,754 lines)
• Topology: 2 spines, 2 ToRs, 3 hosts (single/dual/orphan)
• Tests complete MAC lifecycle, flag transitions, protocol validation
• Production-ready for CI/CD pipelines

Commit 10: Skip Kernel Test (NEW)

Files: tests/topotests/bgp_evpn_mh_l2l3vni_ext_learn/test_evpn_mh_l2l3vni_ext_learn.py
Lines: +10, -1 (~11 lines changed)

• Add pytest.mark.skip to kernel-dependent test suite
• Directs users to FPM-based alternative
• Will be removed once kernel patches are merged

Total Changes: ~4,550 lines across zebra core, documentation, and two comprehensive test suites

Usage

Start zebra with the new option:

zebra --kernel-mac-ext-learn

Or in the configuration/systemd service file:

DAEMON_ARGS="--kernel-mac-ext-learn"

Testing

Build Verification

• ✅ All commits compile independently
• ✅ No build warnings or errors
• ✅ Option parsing works correctly

Functional Testing (Commit 8)

• ✅ ES discovery and advertisement validation
• ✅ VTEP peer list accuracy including failure scenarios
• ✅ L2VNI and L3VNI instantiation with proper VRF association
• ✅ MAC protocol state transitions (data plane ↔ control plane)
• ✅ Orphaned host scenarios
• ✅ Dual-attached host failover and recovery
• ✅ Single-attached host operations
• ✅ MAC delete, relearn, and expiry sequences

Test Environment

• Topology: Multi-homed EVPN VXLAN with spine-leaf architecture
• Test suite: tests/topotests/bgp_evpn_mh_l2l3vni_ext_learn/
• Utility library: tests/topotests/lib/bgp_evpn.py

Compatibility

This change is backward compatible:

• Default behavior remains unchanged (option disabled by default)
• Existing deployments are not affected
• Only platforms that explicitly enable this mode will use the new behavior

Benefits

Performance

• Hardware-accelerated MAC learning and aging
• Reduced CPU overhead on control plane
• Better scalability with larger MAC tables

Operational

• Seamless integration with hardware platforms (e.g., SONiC, Broadcom ASICs)
• Maintains full EVPN control plane functionality
• No impact on existing software-based deployments

Architecture

• Clean separation: Hardware handles data plane, Zebra handles control plane
• Kernel aging disabled for extern_learn MACs
• Proper MAC lifecycle management across hardware and software boundaries

Co-authors

• Patrice Brissette pbrisset@cisco.com
• Tamer Ahmed tamerahmed@microsoft.com

---

Target: FRRouting master branch
Type: Feature enhancement (patch series: 8 commits)
Area: zebra, EVPN, VXLAN
Lines Changed: ~2000+ (infrastructure, implementation, docs, tests)
Test Coverage: Comprehensive topotests included

[greptile-apps]

Greptile Summary

This PR implements EVPN VXLAN Multihoming External Mode support in zebra, enabling hardware-based MAC learning and aging for platforms like SONiC/Broadcom ASICs via a new --kernel-mac-ext-learn CLI flag. The feature is backward-compatible and disabled by default.

• Adds --kernel-mac-ext-learn option wired through zebra_router_init, stored in zrouter.zav.kernel_mac_ext_learn, and exposed via zebra_mac_ext_learn_mode() inline accessor.
• Modifies netlink FDB processing to filter RTPROT_ZEBRA and non-EXT_LEARNED entries in ext-learn mode; adjusts MAC lifecycle (sync, delete, expiry) and suppresses ARP/ND neighbor installs.
• Adds a comprehensive topotest suite (bgp_evpn_mh_l2l3vni_ext_learn) and a shared EVPN test library (lib/bgp_evpn.py).

Confidence Score: 4/5

The change is feature-gated behind --kernel-mac-ext-learn and existing deployments are unaffected; previous review iterations resolved the most critical concerns around nhg_id and nda_ext_flags.

The core ext-learn MAC lifecycle (sync, expiry via flush, BGP notification through zebra_evpn_del_local_mac) is coherent and flows correctly. The interaction between the RTPROT_ZEBRA and NTF_EXT_LEARNED drop filters in rt_netlink.c and graceful-restart FDB dump replays is worth a targeted maintainer review before merge.

zebra/rt_netlink.c — the RTPROT_ZEBRA drop filter and NTF_EXT_LEARNED guard interact with graceful-restart FDB replay and deserve a careful second look.

Important Files Changed

Filename	Overview
zebra/rt_netlink.c	Core netlink FDB processing: adds RTPROT_ZEBRA and NTF_EXT_LEARNED filters in ext-learn mode, fixes duplicate nhg_id set, and rewrites MAC netlink encoding for ext-learn mode with NTF_EXT_LEARNED/NDA_EXT_FLAGS.
zebra/zebra_evpn_mac.c	MAC lifecycle management: ext-learn mode gates sync dp-install on LOCAL_INACTIVE, flushes MACs on hold-expiry, and adjusts BGP-ready transitions for peer-proxy entries.
zebra/zebra_neigh.c	FDB update path: in ext-learn mode, VxLAN-sourced entries are returned early and inoperative-interface entries are guarded before calling zebra_vxlan_local_mac_add_update.
zebra/main.c	Adds --kernel-mac-ext-learn as no_argument option (corrected from earlier optional_argument), passes flag through to zebra_router_init.
zebra/zebra_router.h	Adds kernel_mac_ext_learn field to zebra_architectural_values and the zebra_mac_ext_learn_mode() static inline accessor.
zebra/zebra_evpn_mh.c	Promotes zebra_evpn_flush_local_mac from static to extern and adds a forward declaration for zebra_evpn_es_bypass_update_macs to resolve ordering dependencies.
include/linux/rtnetlink.h	Adds RTPROT_HW=193 for hardware-learned MACs; also changes the SPDX license comment from C-style /* */ to C++ style //.
tests/topotests/bgp_evpn_mh_l2l3vni_ext_learn/test_evpn_mh_l2l3vni_ext_learn.py	New 933-line topotest for ext-learn mode; contains a duplicate import of functools.partial.

Sequence Diagram

sequenceDiagram
    participant HW as Hardware ASIC
    participant Kernel as Linux Kernel (Bridge/VxLAN FDB)
    participant NL as Netlink (rt_netlink.c)
    participant ZN as zebra_neigh.c
    participant ZM as zebra_evpn_mac.c
    participant BGP as BGP (EVPN)

    HW->>Kernel: MAC learned (NTF_EXT_LEARNED set)
    Kernel->>NL: RTM_NEWNEIGH (AF_BRIDGE, NDA_PROTOCOL != RTPROT_ZEBRA)
    NL->>NL: Filter: drop if proto==RTPROT_ZEBRA or !NTF_EXT_LEARNED
    NL->>ZN: dplane ctx (nhg_id=0 in ext-learn mode)
    ZN->>ZM: zebra_vxlan_local_mac_add_update()
    ZM->>BGP: zebra_evpn_mac_send_add_to_client()

    Note over ZM,Kernel: Hold timer expiry (ext-learn mode)
    ZM->>Kernel: dplane_local_mac_del() via zebra_evpn_flush_local_mac
    ZM->>BGP: zebra_evpn_mac_send_del_to_client()

    Note over NL,Kernel: Zebra programs remote/static MAC
    BGP->>ZM: MAC update from BGP EVPN
    ZM->>NL: netlink_macfdb_update_ctx()
    NL->>Kernel: RTM_NEWNEIGH (NTF_EXT_LEARNED + NDA_PROTOCOL=RTPROT_ZEBRA)

Loading

Prompt To Fix All With AI

Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
doc/user/zebra.rst:127
Minor grammatical typo: "its" (possessive) should be "it's" (it is) in the option description.

```suggestion
   Signal to zebra that it's operating in MAC external learn mode.
```

### Issue 2 of 4
include/linux/rtnetlink.h:1
The SPDX identifier was changed from a C-style block comment to a C++ single-line comment. This is a vendored Linux kernel UAPI header and upstream uses the `/* */` form. Diverging here makes future upstream syncs noisier, and the `//` style is not valid C89. The cosmetic change also obscures the only intentional line added (`RTPROT_HW`).

```suggestion
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
```

### Issue 3 of 4
tests/topotests/bgp_evpn_mh_l2l3vni_ext_learn/test_evpn_mh_l2l3vni_ext_learn.py:34-40
`functools.partial` is imported twice — once near the top of the stdlib imports and again after `time`. The second import is redundant and should be removed.

```suggestion
from functools import partial
import time

import pytest
import json
import platform
```

### Issue 4 of 4
zebra/rt_netlink.c:4152-4157
The debug message says "Should drop entry" but the code unconditionally returns 0 immediately after — the entry IS dropped. Using "should" implies the action might not happen, which will confuse anyone reading logs. A definitive phrasing is clearer.

```suggestion
	if (zebra_mac_ext_learn_mode() && !(ndm->ndm_flags & NTF_EXT_LEARNED)) {
		if (IS_ZEBRA_DEBUG_KERNEL)
			zlog_debug("        Dropping entry due to missing extern learn, FLAGS = 0x%x",
				   ndm->ndm_flags);
		return 0;
	}
```

_{Reviews (4): Last reviewed commit: "tests: topotests: EVPN VXLAN MH extern l..." | Re-trigger Greptile}

[pbrisset]

@greptile-apps review

[pbrisset]

@greptile-apps review

[pbrisset]

@greptile-apps review

[pbrisset]

I just ran the checkpatch tool again, current rtnetlink content is clean and SPDX is intentionally unchanged.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[pbrisset]

The CI tool check error is a false positive. This PR is ready for final review